This entire field has been formalized but in my experience the key things to worry about are experimenter and subject bias.
Experimenter bias in a survey context means that the survey writer (Erik) has expectations about the likely community answers. This has been clearly demonstrated, as he already has a feeling about what the German survey results will be even though it hasn't been written. Writing an unbiased survey requires very careful wording and is a tough job. If the entire point of the survey is to find out what the community thinks then the survey should be unbiased. A variety of types of subject bias are overcome by taking a random sample. The claim that the survey takers are self selected is overcome by also recording various demographic information and normalizing the number of responses from demographics, or some other kind of filter. You essentially need to employ psychometric techniques in order to verify the construct validity of the survey (that you can really draw those inferences from those questions). Erik's survey, in my opinion, is likely to have low construct validity and should have been created by a blind, relatively unbiased 3rd party instead. Creating a survey in which the subjects are non-self-selected is a practical impossibility. I can think of some software methods that might help but the better solution is to gather rich demographics and then filter. On Wed, Mar 4, 2009 at 1:32 PM, Gregory Kohs <thekoh...@gmail.com> wrote: > *phoebe ayers* phoebe.wiki at gmail.com writes: > > ++++++ > I'm not sure there's any way to get a non-self-selected survey about anything > on the projects due to anonymity concerns. > ++++++ > > I'm a 17-year veteran of implementing professional quantitative survey > research. Self-selection bias is a very complicated study, but there are > some fairly accessible and intuitive techniques one may implement to create > a thoughtful survey of a target population which minimizes self-selection > bias concerns. This allows the stakeholders to focus on the challenge of > deriving meaning from the response data rather than feeling nausea over the > sampling methodology. > > I am willing to give, pro bono, 45 minutes of telephone consulting time to > any Wikimedia Foundation staff member who is attached to this particular > survey project, on the condition that they will be open and attentive to the > possibility that a properly-designed and fairly-executed survey may not > return results that foster their preconceived desires to railroad through a > license migration (which, unfortunately, is my key takeaway from observing > this discussion). > > -- > Gregory Kohs > Cell: 302.463.1354 > _______________________________________________ > foundation-l mailing list > foundation-l@lists.wikimedia.org > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l > _______________________________________________ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l