Re: [Wiki-research-l] Wiki-research-l Digest, Vol 142, Issue 13

[Zach McDowell]

Hi Kerry,

This research project was really conducted to examine a couple different
things at once. This project was designed to provide data for researchers
trying to understand learning with Wikipedia. Wiki Ed's dashboard was
perfect for this because students participation was already were being
tracked, so layering research on top of that data (and access to emailing
them, cohorts of classes, etcetera) made it an ideal setting for
educational research.

As mentioned in the research report, the questions were designed by
multiple researchers from a variety of backgrounds, so there isn't really
"one" research question, but many approaches to trying to understand what
students are learning, including motivations, perceived value, and skills
transfer, among other things.

I think the confusion here might be coming from the fact Wiki Education
works a little differently than WMF's Education Program - there is a lot of
information on how they operate at their website here - https://wikiedu.org/

Wiki Education as an organization isn't a research organization, it is an
educational organization with the dual mission of education and improving
Wikipedia. They funded the research project but I conducted it through
UMass Amherst.

As for terminology, "instructor" is anyone who is an instructor for the
class (sometimes TT faculty,  sometimes adjunct, sometimes grad student,
sometimes librarian). There are usually no additional Wikipedians in the
classroom, unless the instructor has a specific contact that they invited
in, unbenownst to me.

"Program staff" are folks that work for Wiki Education and support the
classes.

In general, Wiki Ed has a "dashboard" and some support staff, but the
students are not "taught" by a Wikipedian.

I hope this helps!

best
Zach

On Tue, Jun 20, 2017, 5:48 PM Kerry Raymond  wrote:

> Thanks, Zach, for the explanation of these things. I realise the goals of
> WikiEd was not to be a research project but achieve other goals in relation
> to Wikipedia, in which it seems to be very successful from your data.
>
>
>
> I suspect part of my confusion relates to the terms “instructor”,
> “faculty”, and “program staff” (noting that terminology in North American
> universities is very different to Australian universities). Can you unpick
> these for me? Are the “instructors” academic staff of a university or
> college, or, as I interpreted it, people who would probably identify as
> Wikipedians who are volunteering to work on the WikiEd program in their
> local area? I assume “faculty” here means “academic staff member” (in my
> world, “faculty” is an organisational unit composed of academic staff and
> non-academic staff). Who are the “program staff”?
>
>
>
> Thanks
>
>
>
> Kerry
>
>
>
>
>
___
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Re: [Wiki-research-l] Wiki-research-l Digest, Vol 142, Issue 13

[Kerry Raymond]

Thanks, Zach, for the explanation of these things. I realise the goals of 
WikiEd was not to be a research project but achieve other goals in relation to 
Wikipedia, in which it seems to be very successful from your data.

 

I suspect part of my confusion relates to the terms “instructor”, “faculty”, 
and “program staff” (noting that terminology in North American universities is 
very different to Australian universities). Can you unpick these for me? Are 
the “instructors” academic staff of a university or college, or, as I 
interpreted it, people who would probably identify as Wikipedians who are 
volunteering to work on the WikiEd program in their local area? I assume 
“faculty” here means “academic staff member” (in my world, “faculty” is an 
organisational unit composed of academic staff and non-academic staff). Who are 
the “program staff”?

 

Thanks

 

Kerry

 

 

___
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Re: [Wiki-research-l] Wiki-research-l Digest, Vol 142, Issue 13

[Zach McDowell]

Hi Kerry,

Thanks for the feedback! I'll reply to a couple of your points below:

One observation I would make is that like many education experiments,  it
> does not control for (what I call) the "highly motivated researcher
> effect". What I've learned from a lifetime of "new ways to teach" is that
> the standard experiment is to parachute in a highly motivated researcher
> into the classroom to introduce the new method, collect data showing
> improved learning, and then advocate for the new method to be rolled out
> more widely. However, rolling out more widely involves taking regular
> teachers (good, bad, and in-between) to learn and apply a new approach, and
> techniques often fail in the face of teacher lack of enthusiasm to learn
> anything new, complaints it makes more demands on teachers to use the new
> method, etc.


This study was a survey of everyone who would participate across 270
courses and 6700 students. We had 90 different instructors that took the
survey as well, many of them were returning instructors (there is data on
that in the instructor survey), but across the 270 courses there was a good
mix (Wiki Ed has data on that, and I could find that if you're really
interested, but our statistical analysis showed that there was no variation
between new and returning instructors for student responses for the
instructors polled - although a different design or question set with that
in mind could produce different results).

I like that suggestion and, in (hopefully) future iterations, we can spend
more thought on that particular piece when designing surveys (like
specifically tracking how many semesters the instructors have been active
with Wiki Ed).


> In this report it says " the program staff provide Wikipedia training and
> expertise so the faculty do not need to have any experience editing" which
> is a big red flag to me. It would be interesting to see the results in an
> experiment where you first train the faculty and then the faculty carry out
> the engagement with students. And then see the results in 3 years time when
> it's a case of "business as usual" rather than "the new thing".
>

New and returning instructors are encouraged to take (and re-take) a pretty
substantial training program (training module completion is listed in the
data set for both instructors and students). Faculty are "trained" but
there is a limit to the hands-on training that one can offer, of course.


> As a general comment, students like the variety of someone new in their
> classroom. Students do tend to learn more from "real world" assignments
> than "lab" assignments because the real world is more complex. However,
> staff and students are often reluctant to have real world assignments
> significantly influence end-of-term marks/grades because of the
> uncontrollable variables in the real world assignment that makes  it
> difficult to assess the relative achievement of the students.


Absolutely - Joe Reagle and I just did a presentation for New Media
Consortium that recommended that you never grade students on "what remains
on Wikipedia" - Wiki Ed has some good suggestions for grading too.


> I would expect editing Wikipedia articles to suffer from this problem as
> each student will be working on different article(s) of different starting
> size and quality and with different levels of involvement and monitoring by
> other Wikipedians. It was not clear to me from the report if students were
> being assessed on this Wikipedia assignment and how important it was to
> their overall mark/grade.


This was something that across 270 different classes we could not control.
Additionally, it was part of the IRB approval that nothing in the study can
affect the grade of the student directly (eg: instructors can't give extra
credit for taking the survey, they couldn't require students to be present
during the focus groups, no grades were reported). This would have been a
violation of FERPA in the US, as well as many rules in Canada. So in the
end, there was no "assessment" - although each student's work can be traced
back to their course and folks can do their own marking and analysis
(whether through computational means or otherwise) and compare it to
overall data if they like.

In the end, I hope you can make use of the data - as I mentioned above and
a few times throughout the report and elsewhere, this research was meant to
intersect with a variety of research questions, and I've done my best to
open up the data so that folks can investigate their own questions (like
looking at quality of work or addressing content gaps) alongside some of
the student data.

best,

Zach


Zachary J. McDowell, PhD
www.zachmcdowell.com

On Tue, Jun 20, 2017 at 8:00 AM, <
wiki-research-l-requ...@lists.wikimedia.org> wrote:

> Send Wiki-research-l mailing list submissions to
> wiki-research-l@lists.wikimedia.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>