Good point, Asaf. Jessie, is there a way to take this into account when compating costs and benefits?
I agree with Gerard that there can be survey or reporting fatigue, though I have yet to hear an IEG grantee complain. The APG application and reporting system seems more extensive and I can see how it can discourage small orgs from APG funding, on the other hand there seem to be issues of cost-effectiveness and outcome reporting with some existing large APG grantees. Perhaps there should be easier an APG process for small orgs and more specific cost and outcome reporting requirements for large orgs. Pine On Aug 1, 2014 8:56 AM, "Asaf Bartov" <abar...@wikimedia.org> wrote: > Another point to consider is that comparing grants that include staff > compensation to grants that do not is necessarily tipping the scales. > Volunteer time is a cost too (though borne by the volunteers themselves and > not by the funder), and ignoring it in cost-benefit analysis will always > give the impression that grants including staff are significantly less > effective, whether or not they truly are. > > It may make sense to ignore it if the funder is only interested in straight > impact-for-dollars; it seems to me that WMF is a funder that cares about > _movement resources_, including volunteer time, and not just dollars out of > its own budget. > > A. > > > On Thu, Jul 31, 2014 at 10:12 PM, Pine W <wiki.p...@gmail.com> wrote: > > > Hi Jessie, > > > > Thanks for the quick reply. > > > > Issue 1 may be challenging to measure even with Wikimetrics. Can we talk > > about this during the Research Hackathon next week if we can set up a > time > > off-list? > > > > Thanks for the info about issue 2. I am grateful to learn that you did an > > evaluation of PEG. It is interesting to compare that evaluation with the > > evaluation of IEG. A number of grantmaking committee members and grantees > > will be at Wikimania and I hope the PED team will introduce themselves > and > > be available to discuss these studies, especially if there is a plenary > > meeting of all the Meta grantmaking committee members who attend > Wikimania. > > > > Thanks very much, > > > > Pine > > On Jul 31, 2014 2:50 PM, "Jessie Wild" <jw...@wikimedia.org> wrote: > > > > > Thanks for listening to the presentation, Pine! > > > > > > There will be a more comprehensive analysis posted on Meta, but in the > > > meantime to answer your questions: > > > > > > > > >> 1. I'm aware that Program Evaluation is examining the outcomes of > > >> conferences this year, and Jamie and I have discussed this in at least > > two > > >> places on Meta. I'm curious about if and how you plan to measure the > > online > > >> impact of conferences; not just what people and groups say they will > do > > in > > >> post-survey conferences, but what they actually do online in > verifiable > > >> ways in the subsequent 3-12 months. > > >> > > > > > > Jaime and I and the others on the Grantmaking team are working together > > on > > > this, and experimenting with some different ways of evaluating the work > > in > > > the few months following the conferences. One way to do this in a small > > > experiment, for example, is to run a cohort of users who received > > Wikimania > > > Scholarships through Wikimetrics at different increments throughout the > > > year following. This is something I have been curious to do for a long > > > time, but never had the tool to do it on an aggregate level! > > > > > > > > >> > > >> 2. You said in your presentation that there is no direct correlation > > >> between grant size and measurable online impact. From the slides at > > around > > >> the 1:13-1:15 minute marks, it looks to me like the correlation is > > >> negative, meaning that smaller grants produced disproportionately more > > >> impact. I can say that within IEG this occurred partly because we had > > some > > >> highly motivated and generous grantees who volunteered a considerable > > >> amount of time to work with modest amounts of money, and I don't think > > we > > >> should expect that level of generosity from all grantees, but I think > > that > > >> grantmaking committees may want (A) to take into account the level of > > >> motivation of grantees, (B) to consider breaking large block grants > into > > >> discrete smaller projects with individual reporting requirements, and > > (C) > > >> for larger grants where there seem to be a lot of problems with > > reporting > > >> and a disappointing level of cost-effectiveness, to be more assertive > > about > > >> tying funding to demonstrated results and reliable, standardized > > reporting > > >> with assistance from WMF. What do you think? > > >> > > >> Well, there are definite outliers, and the slides aggregate by program > > > type rather than by size. So, for example, several of the IEG grants > were > > > much bigger than than the majority of PEG grants. So - not exactly > > negative > > > correlation (at least, we can't definitively say that). > > > > > > I absolutely agree with your (C) suggestion, and your (B) suggestion is > > > very interesting too - we haven't discussed that one. It may be worth > > > considering if there are larger project-based grants. For the annual > plan > > > grants, we have this in terms of quarterly reports (and midpoint > reports > > > for IEG), so we do try to do interventions with grantees if it looks > like > > > they are off-track. As for (A), based on what we saw through our > > > evaluation of IEG[1], motivation is definitely important but the key > > > difference for outlier performance was from those grantees that had > > *specific > > > target audiences* identified, so they knew exactly who they wanted to > be > > > working with and how to reach those people. So, I would want committees > > to > > > take into account grants with a specific target audience or specific > > target > > > topic area (for quality improvements, for example; we saw this for > > > successful outreach in PEG grants[2]). More explicitly on motivation, > > while > > > it is difficult to measure for new grantees, you can see a lot about > > > someone's motivation and creativity based on their past reports if they > > are > > > a returning grantee. I would definitely encourage our committees to > look > > > back on past reports from returning grantees! > > > > > > - Jessie > > > > > > [1] > > > > https://meta.wikimedia.org/wiki/Grants:IEG/Learning/Round_1_2013/Impact > > > [2] https://meta.wikimedia.org/wiki/Grants:PEG/Learning/2013-14 > > > > > > > > > > > > -- > > > > > > *Jessie Wild SnellerGrantmaking Learning & Evaluation * > > > *Wikimedia Foundation* > > > > > > Imagine a world in which every single human being can freely share in > > > the sum of all knowledge. Help us make it a reality! > > > Donate to Wikimedia <https://donate.wikimedia.org/> > > > > > _______________________________________________ > > Wikimedia-l mailing list, guidelines at: > > https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines > > Wikimedia-l@lists.wikimedia.org > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, > > <mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe> > > > > > > -- > Asaf Bartov > Wikimedia Foundation <http://www.wikimediafoundation.org> > > Imagine a world in which every single human being can freely share in the > sum of all knowledge. Help us make it a reality! > https://donate.wikimedia.org > _______________________________________________ > Wikimedia-l mailing list, guidelines at: > https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines > Wikimedia-l@lists.wikimedia.org > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, > <mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe> _______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe>
