Re: [Wikitech-l] Evaluating Google Summer of Code
Thank you for this summary. It is very useful. Would you mind commenting more on lessons learned about the selection of candidates? We need the right now for https://www.mediawiki.org/wiki/Outreach_Program_for_Women On 11/18/2012 05:17 AM, Sumana Harihareswara wrote: Quim Gil is the administrator for our participation in the Outreach Program for Women, so I'll let him discuss how he wants to manage that and how he'll be setting expectations appropriately. :-) As commented in another thread, I have just drafted https://www.mediawiki.org/wiki/Talk:Mentorship_programs I think mandating two mentors per participant is a very good idea, Quim, what do you think? Defaulting to two is ok, but we can be flexible here - in both ways. Sometimes a single mentor can be trustworthy, sometimes not even two will convince us. ;) Just like with candidates, there is no magic formula for mentors (neither for program admins) :) Also: instead of making newbies write fixed proposals with schedules, these OPW internships can follow broader charters and flex to accommodate students' interests and abilities, and it's easier to rescope iteratively to ensure that something useful gets produced by the end of the three months. I like a lot your People are more important than code meme but we need to find a balance here. I agree that the candidate shouldn't spend much time with formalities like defining whether task X will take exactly 2 or 3 weeks. Still, it is good (for everybody) to go through that exercise before presenting a proposal. It filters those excessively lazy or unskilled and it helps the candidate and the mentor to see whether the generic idea survives a first reality check. Still, your concerns are valid, and I have tried to address them in the list of selection criteria. https://www.mediawiki.org/wiki/Talk:Mentorship_programs#Criteria So here is my proposal: there should be in place some plan of action, ideally for every project, in case it goes overlong and doesn't get completed in time. Good idea. Also included in the selection criteria. -- Quim Gil Technical Contributor Coordinator Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Evaluating Google Summer of Code
TL;DR: we're improving at student retention but need to get better at producing something useful at the end of a mentorship period; some suggestions follow. On 08/28/2012 04:44 AM, Finne Boonen wrote: On Tue, Aug 28, 2012 at 9:41 AM, Ryan Lane rlan...@gmail.com wrote: I think you touch the most important point here. I'm one of the main people who manage GSoC for KDE. KDE is the biggest org taking part in GSoC this year in terms of number of students mentored. In addition we are running our own program (Season of KDE) next to it. So in total I'd say we've mentored about 100 students through these programs this year alone. We didn't get there by accident. You know what the main benefit of GSoC is in my opinion? Getting an organisation into the mindset of mentoring - into the mindset of yes we need to train new people because they can do awesome things even if they screw up sometimes on the way there. +1000. I 3 that way of thinking and I'd very much like our community to have the same attitude towards GSoC. We're a community working towards education of the world. Participating in GSoC as a mentoring organization is a right fit. Agreed. GSoC imho is not about getting code written but getting people involved in our community. As such we should look and see how many people remain involved after GSoC. Finne It has now been three months since the pencils down late in late August when GSoC 2012 officially ended. Therefore it's a good time to look at the nine students we accepted to see how many are still involved with MediaWiki, how many of the GSoC projects themselves are anywhere near what their original targets were, and whether we would prefer to try this again next year or go for a different approach to outreach and mentoring. The answers: * seven of the nine students are still actively working on MediaWiki development. This is better than last year -- at this time last year, only about 2-3 of the eight accepted students remained involved. * one project hit its target (including merge into trunk) and about four others are partially merged into trunk and still have momentum towards completion. This is a little better than last year; about 2 projects from last year are fully merged into trunk and about 1 more is partially merged but there's no momentum towards the finish. https://www.mediawiki.org/wiki/Summer_of_Code_2012/management reflects how I wanted to run this year's GSoC. I think we did well at People are more important than code selection, which is one reason we're doing well at retention. But I failed at ensuring that students were scoping their proposals at about 6 weeks of coding work and dedicating the entire last month of the summer to bugfixing and code review. The proposals usually allotted either no time or about 2 weeks for merging with trunk, pre-deploy code review, and integration. I think smaller scoping would have helped, so in the future, we should simply not accept proposals that do not allot at least a third of the summer to code review and response to code review. I also failed at staying hands-on enough in ensuring frequent mentor-student communication. Having multiple hands-on mentors who were available to do code review REALLY helps, as we see from Nischay Nahata's success (he is our only student this year who got all his project code merged), and I think that in future years we should mandate that all students have two mentors, each of whom commit to reviewing new patchsets within 2 business days. Students can't afford delays in code review. The WMF is improving how it makes time for its engineers to mentor newer contributors (more on that this week) and that will help with this; we're also growing experienced volunteer developers who are interested in mentoring, so I think a 2-mentors-per-student mandate is doable. We should also mandate twice-weekly voice or videocall meetings and do spot checks to ensure they're happening; multiple students didn't have enough communication with their mentors and were shy about pushing for it. Now, to discuss future mentoring approaches: I tried to get us into UCOSP again for fall 2012 and was told they were out of slots, so I'm pushing to get us into UCOSP for spring 2013. We couldn't muster enough mentors to do Google Code-In well so we aren't doing it this year. Quim Gil is the administrator for our participation in the Outreach Program for Women, so I'll let him discuss how he wants to manage that and how he'll be setting expectations appropriately. :-) I think mandating two mentors per participant is a very good idea, Quim, what do you think? Also: instead of making newbies write fixed proposals with schedules, these OPW internships can follow broader charters and flex to accommodate students' interests and abilities, and it's easier to rescope iteratively to ensure that something useful gets produced by the end of the three months. The fact that some of the internships will be in marketing
Re: [Wikitech-l] Evaluating Google Summer of Code
On Tue, Aug 28, 2012 at 3:07 AM, MZMcBride z...@mzmcbride.com wrote: Furthermore, considering GSoC solely in terms of benefit to Mediawiki/Wikipedia is short-sighted. Take a look at the organizations participating: http://www.google-melange.com/gsoc/projects/list/google/gsoc2012 . What would your opinion be if WMF were not on that list? Personally, I don't care very much about being a participant for the sake of being a participant (and I imagine many others feel similarly). I think for a lot of people who watch these Summer of Code projects, it _is_ about benefit to MediaWiki/Wikimedia, particularly as getting involved in these projects can detract from already painfully finite mentoring and reviewing resources. I think you touch the most important point here. I'm one of the main people who manage GSoC for KDE. KDE is the biggest org taking part in GSoC this year in terms of number of students mentored. In addition we are running our own program (Season of KDE) next to it. So in total I'd say we've mentored about 100 students through these programs this year alone. We didn't get there by accident. You know what the main benefit of GSoC is in my opinion? Getting an organisation into the mindset of mentoring - into the mindset of yes we need to train new people because they can do awesome things even if they screw up sometimes on the way there. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Community Communications for Wikidata Wikimedia Deutschland e.V. Obentrautstr. 72 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Evaluating Google Summer of Code
I think you touch the most important point here. I'm one of the main people who manage GSoC for KDE. KDE is the biggest org taking part in GSoC this year in terms of number of students mentored. In addition we are running our own program (Season of KDE) next to it. So in total I'd say we've mentored about 100 students through these programs this year alone. We didn't get there by accident. You know what the main benefit of GSoC is in my opinion? Getting an organisation into the mindset of mentoring - into the mindset of yes we need to train new people because they can do awesome things even if they screw up sometimes on the way there. +1000. I 3 that way of thinking and I'd very much like our community to have the same attitude towards GSoC. We're a community working towards education of the world. Participating in GSoC as a mentoring organization is a right fit. - Ryan ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Evaluating Google Summer of Code
On Tue, Aug 28, 2012 at 9:41 AM, Ryan Lane rlan...@gmail.com wrote: I think you touch the most important point here. I'm one of the main people who manage GSoC for KDE. KDE is the biggest org taking part in GSoC this year in terms of number of students mentored. In addition we are running our own program (Season of KDE) next to it. So in total I'd say we've mentored about 100 students through these programs this year alone. We didn't get there by accident. You know what the main benefit of GSoC is in my opinion? Getting an organisation into the mindset of mentoring - into the mindset of yes we need to train new people because they can do awesome things even if they screw up sometimes on the way there. +1000. I 3 that way of thinking and I'd very much like our community to have the same attitude towards GSoC. We're a community working towards education of the world. Participating in GSoC as a mentoring organization is a right fit. Agreed. GSoC imho is not about getting code written but getting people involved in our community. As such we should look and see how many people remain involved after GSoC. Finne -- Maybe you knew early on that your track went from point A to B, but unlike you I wasn't given a map at birth! Alyssa, Chasing Amy ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Evaluating Google Summer of Code
Hi, It seems like some people are saying that getting workable code out of the Google Summer of Code is a relatively minor aspect of the program - maybe the third or fourth most important outcome of it, behind things like finding new developers, getting better at mentoring, teaching software development, etc. I disagree with this view - I think creating useful projects is the most important part of GSoC, and that this aspect of it should be taken much more seriously than it is. That's for a few reasons: 1) It's Google's money that's paying for all of this, and in their listing of goals for GSoC, they do list most of these things in some form, but the #1 goal they list is Create and release open source code for the benefit of all. [1] 2) It's a real waste to not use this resource to get actual improvements to the software, given the (relatively) limited resources that the Wikimedia Foundation and MediaWiki have. GSoC offers free money, and a framework, for getting development done - it's a tremendous resource that should be taken full advantage of. 3) As MZ notes, even from the vantage point of recruiting developers, having unsuccessful projects is a problem - putting all that effort in, with nothing in the end to show for it, can be demoralizing. And that sort of thing can accumulate: if talented young open-source developers are considering doing a GSoC project for Wikimedia, and they look at the track record for the WMF's previous GSoC project, they may draw negative conclusions about it, and maybe about doing MediaWiki development in general. So how can this be improved? Having mentored three successful GSoC projects for the WMF, I have some perspective on this. As MZ also notes, WMF projects so far have tended to be too large in scope for the length of time allotted. But that's not something that's easy to correct - estimating development time is always a challenge, even when the developers involved are experienced and not newbies. And smaller projects tend not to inspire anyone, mentors or students. So here is my proposal: there should be in place some plan of action, ideally for every project, in case it goes overlong and doesn't get completed in time. That can take several forms: a commitment by the mentor or another experienced developer to finish up the project; a commitment by the WMF to fund the student to finish it, if the student turns out to be serious and it's just a simple lack of time that was the issue; a commitment by the WMF to fund someone else to finish it or just a commitment by the student to finish it themselves, on a volunteer basis. The last of those is tricky, because that tends to be the conclusion at the end of uncompleted projects anyway - the student keeps working on it, but that usually only lasts for a few months before the student's school work and/or regular work get in the way, and then the project often falls by the wayside. A commitment on the WMF and/or mentor side would be the ideal thing - and if there's no such guarantee for a specific project, then that should be taken into consideration when deciding on whether to accept it. Of the three projects I mentored, none of them produced something that was fully usable on the final day of GSoC. In all three cases, more work was put in - the first time, by the student, the second two times by me. The post-GSoC work was significant in all cases, but it was still a lot less than it would have been to try to do the project from scratch. It was a comparatively small amount of effort, to get the code to at least beta-level or higher, that ended up making a huge difference. Something like that, from what I've seen, is almost always needed - so it should be factored into the planning. [1] http://www.google-melange.com/document/show/gsoc_program/google/gsoc2012/faqs#goals -Yaron -- WikiWorks · MediaWiki Consulting · http://wikiworks.com ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Evaluating Google Summer of Code
(Splitting this off from John's critique of ConventionExtension.) Hi. MediaWiki has participated in several (Google) Summer of Code iterations now (https://www.mediawiki.org/wiki/Summer_of_Code) and I'm wondering how this partnership program is evaluated. Whenever this program wraps up at the end of the (Northern Hemisphere's) summer, I always sense a worrying amount of frustration and annoyance from all parties involved. The projects are usually overly large and complex and from what I understand, nearly all of the projects from Google Summer of Code don't end up in production environments. If the projects are lucky, they end up in a MediaWiki extension; if they're unlucky, they rot away in a code repo branch somewhere or behind a configuration variable set to false by default. The end result being that: * the people who worked on these projects are frustrated and annoyed because they didn't get their code deployed [to Wikimedia wikis, a wide audience, or anyone at all in some cases]; * the people who mentored these students are frustrated and annoyed for similar reasons; and * the people (end-users) who wanted to see these projects successfully completed are frustrated and annoyed that these features still don't exist. So I'm left wondering how the cost v. benefit equation works out for this program. How do you evaluate the program and whether MediaWiki ought to remain a continued participant? And, of course, should MediaWiki decide not to participate in Google Summer of Code in 2013, are there other [better] ideas for getting people involved in MediaWiki development? MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Evaluating Google Summer of Code
MZMcBride wrote: MediaWiki has participated in several (Google) Summer of Code iterations now (https://www.mediawiki.org/wiki/Summer_of_Code) and I'm wondering how this partnership program is evaluated. After I posted this, Sumana pointed out that MaxSem has done a great evaluation here: https://www.mediawiki.org/wiki/User:MaxSem/GSoC_analysis. There's also a good overview of past projects at https://www.mediawiki.org/wiki/Summer_of_Code_Past_Projects. And, of course, should MediaWiki decide not to participate in Google Summer of Code in 2013, are there other [better] ideas for getting people involved in MediaWiki development? Perhaps a better model: https://www.mediawiki.org/wiki/UCOSP_Spring_2012? MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Evaluating Google Summer of Code
Most software projects fail (for some definition of fail). Even for highly skilled and highly experienced companies and shops, most software projects fail. I'm not going to look up the Gartner and Forrester and Chaos reports this late on a Monday night, but google away. GSoC is an investment that is not intended to have a short-term payoff. The fact that ANY GSoC code makes to production is fantastic. GSoC is an investment in the long term. It is intended to provide real concrete experience to promising students in real environments, including all the frustrations and annoyances that everyone on a software team experiences in the real world all the time. Schools simply do not provide that experience. Some fraction of those participants will take those experiences into the future of software development, to make real improvements, both to code and to process. Furthermore, considering GSoC solely in terms of benefit to Mediawiki/Wikipedia is short-sighted. Take a look at the organizations participating: http://www.google-melange.com/gsoc/projects/list/google/gsoc2012 . What would your opinion be if WMF were not on that list? On Mon, Aug 27, 2012 at 5:32 PM, MZMcBride z...@mzmcbride.com wrote: (Splitting this off from John's critique of ConventionExtension.) Hi. MediaWiki has participated in several (Google) Summer of Code iterations now (https://www.mediawiki.org/wiki/Summer_of_Code) and I'm wondering how this partnership program is evaluated. Whenever this program wraps up at the end of the (Northern Hemisphere's) summer, I always sense a worrying amount of frustration and annoyance from all parties involved. The projects are usually overly large and complex and from what I understand, nearly all of the projects from Google Summer of Code don't end up in production environments. If the projects are lucky, they end up in a MediaWiki extension; if they're unlucky, they rot away in a code repo branch somewhere or behind a configuration variable set to false by default. The end result being that: * the people who worked on these projects are frustrated and annoyed because they didn't get their code deployed [to Wikimedia wikis, a wide audience, or anyone at all in some cases]; * the people who mentored these students are frustrated and annoyed for similar reasons; and * the people (end-users) who wanted to see these projects successfully completed are frustrated and annoyed that these features still don't exist. So I'm left wondering how the cost v. benefit equation works out for this program. How do you evaluate the program and whether MediaWiki ought to remain a continued participant? And, of course, should MediaWiki decide not to participate in Google Summer of Code in 2013, are there other [better] ideas for getting people involved in MediaWiki development? MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Evaluating Google Summer of Code
Chris McMahon wrote: Most software projects fail (for some definition of fail). Even for highly skilled and highly experienced companies and shops, most software projects fail. I'm not going to look up the Gartner and Forrester and Chaos reports this late on a Monday night, but google away. GSoC is an investment that is not intended to have a short-term payoff. The fact that ANY GSoC code makes to production is fantastic. GSoC is an investment in the long term. It is intended to provide real concrete experience to promising students in real environments, including all the frustrations and annoyances that everyone on a software team experiences in the real world all the time. Schools simply do not provide that experience. Some fraction of those participants will take those experiences into the future of software development, to make real improvements, both to code and to process. Thank you for this. You make some good points. So I guess it might just be a matter of better managing expectations of all parties involved? The disappointment factor is serious and dangerous, in my opinion. It can be mitigated by setting appropriate expectations for the students involved, the mentors involved, and the end-users involved, many of whom desperately want to see some of these features live. Furthermore, considering GSoC solely in terms of benefit to Mediawiki/Wikipedia is short-sighted. Take a look at the organizations participating: http://www.google-melange.com/gsoc/projects/list/google/gsoc2012 . What would your opinion be if WMF were not on that list? Personally, I don't care very much about being a participant for the sake of being a participant (and I imagine many others feel similarly). I think for a lot of people who watch these Summer of Code projects, it _is_ about benefit to MediaWiki/Wikimedia, particularly as getting involved in these projects can detract from already painfully finite mentoring and reviewing resources. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
