Re: [Wiki-research-l] Grant Proposal: Request for Feedback (Response to Aaron Shaw)

[Pine W]

I'm just wondering if this survey could be combined with an upcoming WMF
survey, and combined with the survey I proposed regarding female editors on
projects other than English Wikipedia.

Pinging Tilman and Jaime to ask for their comments about those suggestions.

Thanks,

Pine
On Apr 12, 2015 12:51 PM, "WereSpielChequers" 
wrote:

> Dear Christina,
>
> 1 are you defining your super editors by total or recent edits? Whilst we
> have pretty good editor retention amongst high edit count editors, even
> amongst those with over a 100,000 edits there are inactive and semi active
> editors.
>
> 2 how are you going to ensure that talkpage invites are only responded to
> by the targeted editors?
>
> 3 have you considered emailing your survey? Yes that loses you at least
> the 30% who haven't set an email, but you are much more likely to get your
> responses from the intended target group, also it is quite an effective way
> to contact the inactive and former editors who might not see a talkpage
> note.
>
> 4 What are you going to do to avoid trying to survey deceased Wikipedians?
> Especially with talkpage notes.
>
> 5 how does one make requests to add other questions to your survey?
>
> 6 you mention using census categories to ask the ethnicity question, may
> one ask whose census, Australia, Canada, India, the UK or the USA? Also are
> you intending to replicate the census questions or base your questions
> literally on the census categories generated from those questions?
>
> Regards
>
> Jonathan Cardy
>
>
> On 12 Apr 2015, at 20:49, Christina Shane-Simpson <
> christinam.sh...@gmail.com> wrote:
>
> Hello Aaron and Other Wiki Researchers,
>
> Thank you for responding so quickly and thoroughly to my recent proposal!
> Many of your concerns align with issues I’ve been discussing with my
> research team, so I’m glad to hear that we’re overlapping in that sense.
> Apologies in advance for the length of the following:
>
> - - Sampling:  I completely agree with your concerns in response
> to the (relatively) recent revisit to the original Gender Gap results.  As
> an exploratory study, I don’t think we could accurately represent the
> entire Wikipedia community or make causal inferences about the community as
> a whole due to the voluntary nature of the survey and the potential for
> inaccuracies in self-reporting.  However, I’m hoping that this preliminary
> project could reveal a few new patterns that might be explored in greater
> depth at a later date.
>
> Based on the Wikipedia editor rankings, I’d planned to pull the top 20% of
> editors and post on their Talk Pages, giving us the “super-editor” sample.
> Since the two remaining samples are more difficult to recruit, I’m
> currently exploring the most effective way to obtain a randomized sample of
> the active (moderate) and inactive editors (infrequent edits) – this will
> likely be developed with the assistance of someone more skilled in
> programming than myself.  I’ve also been speaking with a statistician about
> alternative methods, beyond propensity-matching, where we might account for
> response biases that are likely to occur.  However, I’d be very open to
> suggestions from this community about effectively sampling from Wikipedia
> and methods you’ve used to account for biases common in these surveys.
>
> - - Self-Report Measures of Edit History:  This would only serve
> to verify the editor ranking and provide a more thorough context by which
> the editor feels he/she makes contributions to the Wikipedia community.
> Since we’ll have usernames – via Talk Pages – as you suggested, I’d like to
> explore actual editing behaviors given that we’d have the resources to do
> so.
>
> -- Collaboration:  Participant fatigue is a huge concern with all
> of these online surveys targeting active editors.  I believe you’re correct
> that the WMF is planning another editor survey, but I had hoped to provide
> some foundation for other themes that might be explored in these larger
> surveys.  The prior WMF surveys didn’t provide as much depth as we might
> need to reveal any patterns in editing behaviors.  I’ve also reached out to
> a couple of other proposals, with similar interests, to determine whether
> we can compliment each other’s efforts.  I think these types of
> collaborations are very do-able and may help us to limit the frequency of
> Wikipedia editor surveys.
>
> - Missing Measures and People:  I was able to access your
> article, so thank you for linking it!  I’ve been reviewing the literature
> to clarify variables (such as the *web use* you identify) to determine
> which should be included in the survey.  In order to keep the survey at a
> reasonable length, I’d hoped to capture some of these editing barriers via
> themes captured in the open-ended responses.  This might be particularly
> relevant in the context of editors’ *perceived* barriers, which might
> vary based on the aforementioned traits.  However, I agree 

Re: [Wiki-research-l] Grant Proposal: Request for Feedback (Response to Aaron Shaw)

[WereSpielChequers]

Dear Christina,

1 are you defining your super editors by total or recent edits? Whilst we have 
pretty good editor retention amongst high edit count editors, even amongst 
those with over a 100,000 edits there are inactive and semi active editors.

2 how are you going to ensure that talkpage invites are only responded to by 
the targeted editors? 

3 have you considered emailing your survey? Yes that loses you at least the 30% 
who haven't set an email, but you are much more likely to get your responses 
from the intended target group, also it is quite an effective way to contact 
the inactive and former editors who might not see a talkpage note.

4 What are you going to do to avoid trying to survey deceased Wikipedians? 
Especially with talkpage notes.

5 how does one make requests to add other questions to your survey?

6 you mention using census categories to ask the ethnicity question, may one 
ask whose census, Australia, Canada, India, the UK or the USA? Also are you 
intending to replicate the census questions or base your questions literally on 
the census categories generated from those questions?

Regards

Jonathan Cardy


> On 12 Apr 2015, at 20:49, Christina Shane-Simpson 
>  wrote:
> 
> Hello Aaron and Other Wiki Researchers,
> 
> Thank you for responding so quickly and thoroughly to my recent proposal!  
> Many of your concerns align with issues I’ve been discussing with my research 
> team, so I’m glad to hear that we’re overlapping in that sense.  Apologies in 
> advance for the length of the following:
> 
> - - Sampling:  I completely agree with your concerns in response to 
> the (relatively) recent revisit to the original Gender Gap results.  As an 
> exploratory study, I don’t think we could accurately represent the entire 
> Wikipedia community or make causal inferences about the community as a whole 
> due to the voluntary nature of the survey and the potential for inaccuracies 
> in self-reporting.  However, I’m hoping that this preliminary project could 
> reveal a few new patterns that might be explored in greater depth at a later 
> date.
> 
> Based on the Wikipedia editor rankings, I’d planned to pull the top 20% of 
> editors and post on their Talk Pages, giving us the “super-editor” sample.  
> Since the two remaining samples are more difficult to recruit, I’m currently 
> exploring the most effective way to obtain a randomized sample of the active 
> (moderate) and inactive editors (infrequent edits) – this will likely be 
> developed with the assistance of someone more skilled in programming than 
> myself.  I’ve also been speaking with a statistician about alternative 
> methods, beyond propensity-matching, where we might account for response 
> biases that are likely to occur.  However, I’d be very open to suggestions 
> from this community about effectively sampling from Wikipedia and methods 
> you’ve used to account for biases common in these surveys.  
> 
> - - Self-Report Measures of Edit History:  This would only serve to 
> verify the editor ranking and provide a more thorough context by which the 
> editor feels he/she makes contributions to the Wikipedia community.  Since 
> we’ll have usernames – via Talk Pages – as you suggested, I’d like to explore 
> actual editing behaviors given that we’d have the resources to do so.
> 
> -- Collaboration:  Participant fatigue is a huge concern with all of 
> these online surveys targeting active editors.  I believe you’re correct that 
> the WMF is planning another editor survey, but I had hoped to provide some 
> foundation for other themes that might be explored in these larger surveys.  
> The prior WMF surveys didn’t provide as much depth as we might need to reveal 
> any patterns in editing behaviors.  I’ve also reached out to a couple of 
> other proposals, with similar interests, to determine whether we can 
> compliment each other’s efforts.  I think these types of collaborations are 
> very do-able and may help us to limit the frequency of Wikipedia editor 
> surveys.
> 
> - Missing Measures and People:  I was able to access your article, so 
> thank you for linking it!  I’ve been reviewing the literature to clarify 
> variables (such as the web use you identify) to determine which should be 
> included in the survey.  In order to keep the survey at a reasonable length, 
> I’d hoped to capture some of these editing barriers via themes captured in 
> the open-ended responses.  This might be particularly relevant in the context 
> of editors’ perceived barriers, which might vary based on the aforementioned 
> traits.  However, I agree that the study would likely benefit form some 
> further questioning about editing experiences and I’ll be adding this into 
> the proposal.
> 
>   - Missing People and Sampling:  Your main concern also parallels the 
> concerns of my research team.  I’ve been speaking with my team about 
> potentially recruiting a passive Wikipedia user sample that w

Re: [Wiki-research-l] Grant Proposal: Request for Feedback (Response to Aaron Shaw)

[Christina Shane-Simpson]

Hello Aaron and Other Wiki Researchers,

Thank you for responding so quickly and thoroughly to my recent proposal!  Many 
of your concerns align with issues I’ve been discussing with my research team, 
so I’m glad to hear that we’re overlapping in that sense.  Apologies in advance 
for the length of the following:

- - Sampling:  I completely agree with your concerns in response to the 
(relatively) recent revisit to the original Gender Gap results.  As an 
exploratory study, I don’t think we could accurately represent the entire 
Wikipedia community or make causal inferences about the community as a whole 
due to the voluntary nature of the survey and the potential for inaccuracies in 
self-reporting.  However, I’m hoping that this preliminary project could reveal 
a few new patterns that might be explored in greater depth at a later date.

Based on the Wikipedia editor rankings, I’d planned to pull the top 20% of 
editors and post on their Talk Pages, giving us the “super-editor” sample.  
Since the two remaining samples are more difficult to recruit, I’m currently 
exploring the most effective way to obtain a randomized sample of the active 
(moderate) and inactive editors (infrequent edits) – this will likely be 
developed with the assistance of someone more skilled in programming than 
myself.  I’ve also been speaking with a statistician about alternative methods, 
beyond propensity-matching, where we might account for response biases that are 
likely to occur.  However, I’d be very open to suggestions from this community 
about effectively sampling from Wikipedia and methods you’ve used to account 
for biases common in these surveys.  

- - Self-Report Measures of Edit History:  This would only serve to 
verify the editor ranking and provide a more thorough context by which the 
editor feels he/she makes contributions to the Wikipedia community.  Since 
we’ll have usernames – via Talk Pages – as you suggested, I’d like to explore 
actual editing behaviors given that we’d have the resources to do so.

-- Collaboration:  Participant fatigue is a huge concern with all of 
these online surveys targeting active editors.  I believe you’re correct that 
the WMF is planning another editor survey, but I had hoped to provide some 
foundation for other themes that might be explored in these larger surveys.  
The prior WMF surveys didn’t provide as much depth as we might need to reveal 
any patterns in editing behaviors.  I’ve also reached out to a couple of other 
proposals, with similar interests, to determine whether we can compliment each 
other’s efforts.  I think these types of collaborations are very do-able and 
may help us to limit the frequency of Wikipedia editor surveys.

- Missing Measures and People:  I was able to access your article, so 
thank you for linking it!  I’ve been reviewing the literature to clarify 
variables (such as the web use you identify) to determine which should be 
included in the survey.  In order to keep the survey at a reasonable length, 
I’d hoped to capture some of these editing barriers via themes captured in the 
open-ended responses.  This might be particularly relevant in the context of 
editors’ perceived barriers, which might vary based on the aforementioned 
traits.  However, I agree that the study would likely benefit form some further 
questioning about editing experiences and I’ll be adding this into the proposal.

- Missing People and Sampling:  Your main concern also parallels the 
concerns of my research team.  I’ve been speaking with my team about 
potentially recruiting a passive Wikipedia user sample that would serve as a 
comparison.  It was my original hope that a small incentive would encourage 
even the infrequent editors to complete the survey measure, but in the event 
that they don’t we’ll need that comparison group.  Our greatest barrier would 
be matching the “pertinent” comparison sample characteristics with our 
super-editors.   I’m not sure that we can achieve this yet, but more to come as 
I explore this option.

Thank you again Aaron for your thorough feedback!  As I’ve been following this 
listserv, I’m incredibly grateful that we have developed such a strong 
research-oriented Wikipedia community.  

Sincerely,

Christina

___
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Re: [Wiki-research-l] Grant Proposal: Request for Feedback

[aaron shaw]

Greeting Christina!

Thanks for sharing this and notifying us on the list.

Overall, I am very supportive of additional attempts to do more rigorous
survey research on Wikipedians. Some questions that I think you could try
to address in the proposal:

- *Sampling*: You mention that you plan to stratify your sample based on
past edit history and recruit via talk page messages. However, beyond this
you say nothing about the logistics of subject sampling, recruitment, or
any approaches you will take to address the fact that conducting
representative surveys in online communities is very, very difficult. Can
you elaborate on this aspect of your study? In particular, how will your
approach address shortcomings in data and sample quality that have affected
previous surveys

of Wikipedia contributors?

- *Self-report measures of edit history: *Why ask the respondents to
self-report their edit histories (this kind of thing is notoriously hard to
do accurately) when you could ask them to provide their usernames or at
least link their usernames to their survey responses (since you're
recruiting via talk page messages anyway)?

- *Collaboration w related studies: *There are several other ongoing
efforts to survey wikipedians -- even at least one other one
(link to "official" publication
is gated, but other versions are available for free) focused on social
psychological concerns. Also, my impression is that the WMF is involved in
planning another editor survey in the near future. How will your approach
complement/extend/overlap with these other efforts? Will you make any
effort to collaborate with these ongoing studies? How will your study avoid
subject exhaustion -- especially among more active wikipedians who may find
themselves invited to participate in many surveys?

- *Missing measures and missing people:* Previous studies have shown that a
variety of additional factors may figure in shaping the participation
practices of Wikipedians as well as those who might edit Wikipedia but
choose not to do so. For example, in a recent paper

(again,
gated link, but I am also happy to provide copies to those who would like
access) that I co-wrote w Eszter Hargittai, we find that web use skills
are, in some ways, even more robust predictors of wikipedia contribution
than gender. There are many other examples of important measures that
predict participation in various ways as well, whether it be individual's
trust/caution attitudes, newcomer experiences, etc. Which of these measures
will you include? How will you ensure that you have included the most
important measures in this survey study since survey results are otherwise
quitre prone to omitted variable bias?

*Missing people and sampling on the dependent variable: *Maybe most
importantly, insofar as you say that you are interested in understanding
factors that determine who edits, you are selecting on the dependent
variable (wikipedia editing) by limiting your study to individuals who have
accounts on the encyclopedia and edit already. It strikes me as especially
egregious that you are requiring survey respondents to read and reply to
the survey recruitment materials via talk page message. This means that
precisely those individuals who participate least (and who would provide
your study with necessary variation on the outcome of interest) are the
least likely to respond and to be included in the study. As a result, I
fear that your findings will not speak to these questions effectively
unless you find an alternative method of sampling and recruitment.


I hope that these comments are helpful for you as you continue to refine
the study design. I really think you're pursuing a critical set of concerns
in this study and I am eager to see it succeed in the most effective way
possible!

yours,
Aaron




On Sun, Apr 12, 2015 at 8:44 AM, Christina Shane-Simpson <
christinam.sh...@gmail.com> wrote:

>  Hello Fellow Wiki Researchers,
>
>  I’ve recently posted a project proposal under the Inspire Campaign and
> would love feedback from this community on the research proposal, 
> *Characterization
> of Editors on Wikipedia*:
>
>  In order to accurately explore the main goals of the Inspire Campaign,
> we must be able to effectively characterize our community. Any
> interventions that we develop should reflect and match the needs of the
> target population, requiring a thorough understanding of the traits and
> behaviors of our community of editors. As a direct extension of the recent
> gender gap research on Wikipedia and to explore other potential areas of
> inequality, we’d like to conduct another study that compares the traits of
> the super-editor, the active editor (moderate editing), and the inactive
> editor (infrequent edits).
>
> The proposed project would use an online self-repor