Re: [Wikimedia-l] Timothy Sandole and (apparently) $53, 690 of WMF funding

[Srikanth Ramakrishnan]

Pine, I have another question to add to the initial question:
Will the Foundation prohibit chapters and other thematic organizations from
the "creation of paid roles that have article writing as a core focus,
regardless of who is initiating or managing the process" as a condition of
receiving WMF funding and using the WMF trademarks?

"Will the WMF itself ensure that foundation money will not be used to
generate content on a long term basis?"

I think this is more of an appropriate question?

I have used long term because stuff like Contests/Challenges [there is one
on right now]  can be considered short term, you know, just to keep editors
interests up.



On Wed, May 7, 2014 at 6:24 AM, Pete Forsyth  wrote:

> On Tue, May 6, 2014 at 5:34 PM, Nathan  wrote:
>
> > On Tue, May 6, 2014 at 8:18 PM, Pete Forsyth 
> > wrote:
> > >
> > >
> > > I want to point out something that stands out to me. This is not an
> > > outright contradiction, but it's a puzzling contrast. In an unrelated
> > > thread on this email list, Executive Director Sue Gardner recently
> said:
> > >
> > > "Editorial policies [for WMF staff] are developed, and therefore also
> > > best-understood and best-enforced, not by the WMF but by the
> community."
> > > [1]
> > >
> > > That is the WMF policy as it applies to WMF staff: essentially, no
> > special
> > > rules, use your own judgment in interpreting how to best comply with
> > > community standards. But here, in the report Sue authored, it seems
> there
> > > is a very different standard for movement partners who seek funding or
> > > endorsement from the WMF:
> > >
> > > "In the future, the Wikimedia Foundation will not support or endorse
> the
> > > creation of paid roles that have article writing as a core focus,
> > > regardless of who is initiating or managing the process." [2]
> > >
> > > Again: this is not a direct contradiction, and it is entirely within
> the
> > > rights of the WMF to apply different standards to its own staff vs. to
> > > other organizations. But I do think it deserves some careful
> > consideration,
> > > as to *why* such different standards would be appropriate.
> > >
> > > Decision point #1 in the Belfer Center report is not something that is
> > > based in any Wikipedia policy. It does have a basis in the Wikipedian
> in
> > > Residence page on the Outreach Wiki.[3] That is an important page, and
> I
> > > believe many in the movement consider it to have the weight of a formal
> > > policy; but I don't. Elevating it from a best practice recommendation
> to
> > an
> > > absolute rule is a significant step, and one that I don't believe
> should
> > be
> > > taken lightly.
> >
> >
> > Hi Pete,
> >
> > Perhaps I'm misunderstanding you, and I hope you can clarify for me so
> that
> > I can follow your position. I don't see the contradiction at all between
> > the two policy-related statements. In the first case, the WMF says that
> the
> > editorial policies that apply to its employees are promulgated by
> specific
> > projects and their communities, not the WMF. In the second, it says
> > effectively that the WMF will not sponsor paid editing. The presumption
> in
> > the first instance is that the WMF already does not pay its employees to
> > edit, so Sue was not referring to "paid editing" at all.  Russavia's
> > question was about editing with a conflict of interest, not payment.
> >
> > I'm not seeing any conflict between those two statements, and the WMF
> does
> > not appear to me to be applying different standards to others than to
> > itself. In fact, the only time paid editing by an employee has come up as
> > an issue, the employee was quickly dismissed. Perhaps you can explain?
> > ___
> >
> >
> Nathan:
>
> Again, I don't say it's a contradiction, it's not. But I do think it's an
> important contrast, and yes, I'll try to clarify why.
>
> Does the Wikimedia Foundation create additional policies, related to
> editing Wikipedia, over and above those established by the Wikipedia
> community and documented on Wikipedia?
>
> For its staff, according to the email I quoted above, the answer is "no."
> (You're right, there is one case that might suggest otherwise, relating to
> paid editing -- but we don't, and shouldn't, have public access to all the
> specifics of that case, so it's a tricky one to draw conclusions from,
> especially in a public forum.) But, there are countless ways in which
> Wikimedia Foundation staff edit Wikipedia and other projects as a part of
> their compensated work (and also, in their free time). There is apparently
> no policy from the WMF governing that behavior beyond general trust in its
> staff to abide by community-set rules.
>
> For other organizations, though, that might seek Wikimedia funds and/or
> endorsement, the answer is apparently "yes" (according to the Belfer Center
> report.)
>
> I think that's a contrast that merits some consideration. I think Pine's
> example is a goo

Re: [Wikimedia-l] Cracking Wikipedia

[Russavia]

I have been quite succesful in getting media relicenced for our
projects. My first major success was the Kremlin[1], and in my letter
to them[2] I made it very clear that media in articles would help to
better articulate and present the points of view that should be
present in relevant articles. Today, we have over 11,000 Kremlin
images on Commons, of which over 1/3 are in use across our
projects.[3]

Governmental media, be it the Kremlin or the White House, is somewhat
promoting a "brand". This is especially evident by another government
department which made available its media under a free licence after
my request -- Korea.net.[4] Korea.net is operated by the Korean
Culture and Information Service whose mission it is to "introduce"
Korean culture to the world and to "uplift" the national image of
South Korea.[5] Their images are now widely in use on Korean
Wikipedia, and, disappointingly, only to a limited extent on English
Wikipedia and other projects.[6]

Whilst it is often unspoken/unwritten that media can have obvious
benefits, the New Zealand government is different and clearly
states:[7]

"It is widely recognised that re-use of this material by individuals
and organisations may have significant creative and economic benefit
for New Zealand."

And sure enough, the New Zealand Defence Force has put into place
NZGOAL recommendations, and we happily accept their media,[8] put them
into use on our projects.[9]

In the last 12 months or so, I have also started to target companies
to relicence their media. I began to do this after seeing that LG
Electronics have for a long time made available their extensive
photograph library under a Creative Commons licence.[10] And, of
course, we happily accept their media on our projects[11], and put
them into use.[12] Some of their photos are in use on projects, and
the "product placement" is clear.[13]

I have so far managed to get Maersk Line,[14] SuperJet
International,[15] Austrian Airlines,[16] Jetstar Airways,[17] amongst
others to provide their media under a free licence. And I am
discussing with a range of other companies in getting them to provide
media under a free licence. Many of these images are in use on
projects, and some of them have been recognised on our projects at
featured photos.[18] And some commercial providers of content are such
that we might not have media otherwise.[19]

The "Pirelli" video[20] obviously does not demonstrate that Pirelli
has done anything on Wikipedia, or even that it intends to. For all we
know, it is an initial pitch to Pirelli on how an advertising company
could help to promote Pirelli on Wikipedia -- they have are on point
on some things, but way off on others. But already we have editors
being silly and calling for regulation on how images should be used,
and suggesting that editors who upload images promoting brands should
be shown the door and the images deleted.[21] Luckily, there are some
cooler, and more reasonable, heads in such discussions.[22]

Regardless of what is going on with Pirelli, if this is an avenue that
they wish to pursue, then I would welcome their media contributions,
and I would also welcome the opportunity to consult, for a reasonable
fee, on how best to provide media to our projects and enter into a
win-win relationship for both us and them.

Russavia


[1] https://commons.wikimedia.org/wiki/Template:Kremlin.ru
[2] https://en.wikipedia.org/wiki/User:Russavia/letter
[3] http://bit.ly/1jbZDNv
[4] 
https://commons.wikimedia.org/wiki/Category:Files_from_Korea.net_Flickr_stream
[5] 
https://en.wikipedia.org/wiki/Korean_Culture_and_Information_Service#Overview
[6] http://bit.ly/Od0lx6
[7] http://www.nzdf.mil.nz/news/image-galleries/creative-commons/default.htm
[8] 
https://commons.wikimedia.org/wiki/Category:Files_from_New_Zealand_Defence_Force_Flickr_stream
[9] http://bit.ly/RquHOu
[10] https://www.flickr.com/photos/lge
[11] https://commons.wikimedia.org/wiki/Category:Images_from_LGEPR
[12] http://bit.ly/1nnxinR
[13] 
https://commons.wikimedia.org/wiki/File:Eva_Longoria_Parker_and_Victoria_Beckham.jpg
and https://commons.wikimedia.org/wiki/File:Eva_Longoria_Parker_crop.jpg
for example
[14] 
https://commons.wikimedia.org/wiki/Category:Files_from_Maersk_Line_Flickr_stream
[15] 
https://commons.wikimedia.org/wiki/Category:Files_from_SuperJet_International_Flickr_stream
[16] 
https://commons.wikimedia.org/wiki/Category:Photographs_by_Austrian_Airlines
[17] https://commons.wikimedia.org/wiki/Category:Photographs_by_Jetstar_Airways
[18] 
https://commons.wikimedia.org/wiki/File:Air-to-air_photo_of_a_Sukhoi_Superjet_100_%28RA-97004%29_over_Italy.jpg
and 
https://commons.wikimedia.org/wiki/File:Computer_generated_image_of_the_M%C3%A6rsk_Triple_E_Class_(1).jpg
[19] https://commons.wikimedia.org/wiki/Category:Photographs_by_Uri_Tours
- 
https://commons.wikimedia.org/wiki/File:Participants_in_the_2012_Pyongyang_Marathon_running_past_the_Arch_of_Triumph.jpg
for example
[20] https://www.youtube.com/watch?v=0Nfbt

[Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Anthony Cole]

Could someone please point me to all the studies the WMF have conducted
into the reliability of Wikipedia's content? I'm particularly interested in
the medical content, but would also like to look over the others too.
Cheers.

Anthony Cole 
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Risker]

On 7 May 2014 16:17, Anthony Cole  wrote:

> Could someone please point me to all the studies the WMF have conducted
> into the reliability of Wikipedia's content? I'm particularly interested in
> the medical content, but would also like to look over the others too.
> Cheers.
>
> Anthony Cole 
>


I've often thought about this myself, and I'm fairly certain the WMF has
never done any serious assessment of article quality. Different projects
have done so on their own, through content auditing processes and the
development of Wikipedia 1.0, but that affects a minority of articles.
There are some real challenges in coming up with workable metrics.

For example - Is a stub article inaccurate, incomplete, or really contains
all the information it's likely ever going to get?

How does one assess the accuracy of articles where there are multiple
sources that we'd consider reliable, but who provide contradictory
information on a topic?  That would include, for example, all the ongoing
boundary issues involving multiple countries, the assessment of historical
impact of certain events or persons, and certain scientific topics where
new claims and reports happen fairly frequently and may or may not have
been reproduced.  There may also be geographic or cultural factors that
affect the quality of an article, or the perceived notability of a subject,
and challenges dealing with cross-language reference sources.

Many of the metrics used  for determining "quality" in audited articles on
English Wikipedia have very little to do with the actual quality of the
article.  From the perspective of providing good information, a lot of
Manual of Style practices are nice but not required.  Certain accessibility
standards (alt text for images, media positioning so as not to adversely
affect screen-readers) are not quality metrics, strictly speaking; they're
*accessibility* standards.  There remains a huge running debate about
whether or not infoboxes should be required, what information should be in
them, how to deal with controversial or complex information in infoboxes,
etc.

So I suppose the first step would be in determining what metrics should be
included in a quality assessment of a project.

Risker/Anne
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

Anne, there are really well-established systems of scholarly peer review.
There is no need to reinvent the wheel, or add distractions such as
infoboxes and other bells and whistles.

I find it extraordinary that, after 13 years, a project designed to make
the sum of human knowledge available to humanity, with an annual budget of
$50 million, has no clue how to measure the quality of the content it is
providing, no apparent interest in doing so, and no apparent will to spend
money on it.

For what it's worth, there was a recent external study of Wikipedia's
medical content that came to unflattering results:

http://www.jaoa.org/content/114/5/368.full

---o0o---

Most Wikipedia articles for the 10 costliest conditions in the United
States contain errors compared with standard peer-reviewed sources. Health
care professionals, trainees, and patients should use caution when using
Wikipedia to answer questions regarding patient care.

Our findings reinforce the idea that physicians and medical students who
currently use Wikipedia as a medical reference should be discouraged from
doing so because of the potential for errors.

---o0o---


On Wed, May 7, 2014 at 10:59 PM, Risker  wrote:

> On 7 May 2014 16:17, Anthony Cole  wrote:
>
> > Could someone please point me to all the studies the WMF have conducted
> > into the reliability of Wikipedia's content? I'm particularly interested
> in
> > the medical content, but would also like to look over the others too.
> > Cheers.
> >
> > Anthony Cole 
> >
>
>
> I've often thought about this myself, and I'm fairly certain the WMF has
> never done any serious assessment of article quality. Different projects
> have done so on their own, through content auditing processes and the
> development of Wikipedia 1.0, but that affects a minority of articles.
> There are some real challenges in coming up with workable metrics.
>
> For example - Is a stub article inaccurate, incomplete, or really contains
> all the information it's likely ever going to get?
>
> How does one assess the accuracy of articles where there are multiple
> sources that we'd consider reliable, but who provide contradictory
> information on a topic?  That would include, for example, all the ongoing
> boundary issues involving multiple countries, the assessment of historical
> impact of certain events or persons, and certain scientific topics where
> new claims and reports happen fairly frequently and may or may not have
> been reproduced.  There may also be geographic or cultural factors that
> affect the quality of an article, or the perceived notability of a subject,
> and challenges dealing with cross-language reference sources.
>
> Many of the metrics used  for determining "quality" in audited articles on
> English Wikipedia have very little to do with the actual quality of the
> article.  From the perspective of providing good information, a lot of
> Manual of Style practices are nice but not required.  Certain accessibility
> standards (alt text for images, media positioning so as not to adversely
> affect screen-readers) are not quality metrics, strictly speaking; they're
> *accessibility* standards.  There remains a huge running debate about
> whether or not infoboxes should be required, what information should be in
> them, how to deal with controversial or complex information in infoboxes,
> etc.
>
> So I suppose the first step would be in determining what metrics should be
> included in a quality assessment of a project.
>
> Risker/Anne
> ___
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[David Gerard]

On 7 May 2014 23:14, Andreas Kolbe  wrote:

> For what it's worth, there was a recent external study of Wikipedia's
> medical content that came to unflattering results:
> http://www.jaoa.org/content/114/5/368.full


Osteopaths.

Perhaps we could ask the chiropractors and homeopaths what they think too.


- d.

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Risker]

On 7 May 2014 18:14, Andreas Kolbe  wrote:

> Anne, there are really well-established systems of scholarly peer review.
> There is no need to reinvent the wheel, or add distractions such as
> infoboxes and other bells and whistles.
>
> I find it extraordinary that, after 13 years, a project designed to make
> the sum of human knowledge available to humanity, with an annual budget of
> $50 million, has no clue how to measure the quality of the content it is
> providing, no apparent interest in doing so, and no apparent will to spend
> money on it.
>
> For what it's worth, there was a recent external study of Wikipedia's
> medical content that came to unflattering results:
>
> http://www.jaoa.org/content/114/5/368.full
>
> ---o0o---
>
> Most Wikipedia articles for the 10 costliest conditions in the United
> States contain errors compared with standard peer-reviewed sources. Health
> care professionals, trainees, and patients should use caution when using
> Wikipedia to answer questions regarding patient care.
>
> Our findings reinforce the idea that physicians and medical students who
> currently use Wikipedia as a medical reference should be discouraged from
> doing so because of the potential for errors.
>
>

> Doesn't help very much in assessing the quality of the article on
> [[Liancourt Rocks]] - when depending on where in the world one is, the
> article can be reasonably accurate or completely inaccurate.  This is one
> of the geographic issues of which I speak.


There are also issues with the study you reference - it's quite biased
toward American information and the articles only have two reviewers. It
perhaps points out how easy it is to get junk science published in
peer-reviewed journals if the topic is "sexy" enough - their own
study wouldn't meet our standards for inclusion.

Risker/Anne
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Nathan]

On Wed, May 7, 2014 at 6:19 PM, David Gerard  wrote:

> On 7 May 2014 23:14, Andreas Kolbe  wrote:
>
> > For what it's worth, there was a recent external study of Wikipedia's
> > medical content that came to unflattering results:
> > http://www.jaoa.org/content/114/5/368.full
>
>
> Osteopaths.
>
> Perhaps we could ask the chiropractors and homeopaths what they think too.
>
>
> - d.


You misunderstand - these are doctors of osteopathic medicine in the U.S.
They are effectively the equivalent of typical medical doctors. The term
osteopath as you use it in the UK and elsewhere has a very different
meaning here.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

"In a blinded process, we randomly selected 10 reviewers to examine 2 of
the selected Wikipedia articles. Each reviewer was an internal medicine
resident or rotating intern at the time of the assignment. This arrangement
created redundancy, giving the study 2 independent reviewers for each
article. Also, by using physicians as reviewers, we ensured a baseline
competency in medical literature interpretation and research."

The articles reviewed were coronary artery disease, lung cancer, major
depressive disorder, osteoarthritis, chronic obstructive pulmonary disease,
hypertension, diabetes mellitus, back pain, hyperlipidemia and concussion.

Carry on.




On Wed, May 7, 2014 at 11:19 PM, David Gerard  wrote:

> On 7 May 2014 23:14, Andreas Kolbe  wrote:
>
> > For what it's worth, there was a recent external study of Wikipedia's
> > medical content that came to unflattering results:
> > http://www.jaoa.org/content/114/5/368.full
>
>
> Osteopaths.
>
> Perhaps we could ask the chiropractors and homeopaths what they think too.
>
>
> - d.
>
> ___
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Michael Maggs]

Measuring the quality of Wikipedia articles in general is an issue that 
Wikimedia UK is interested in looking at, though by means of automation rather 
than the gold-standard but much less scalable method of scholarly peer review.

Our early-stage plans for a large-scale IT project to provide automated 
quality-measuring tools for the community can be found at [1].

On the corresponding talk page my fellow trustee Simon Knight has recently 
posted the results of a literature survey that may be of some interest, 
although again he was focusing on automation rather than individual manual 
review. Almost all of the research has been done by academics, and very little 
seems to have found its way back to the Wikimedia space where it could be 
applied in practice.

Michael

[1]  
https://wikimedia.org.uk/wiki/Technology_Committee/Project_requests/WikiRate_-_rating_Wikimedia



On 7 May 2014, at 23:14, Andreas Kolbe  wrote:

> Anne, there are really well-established systems of scholarly peer review.
> There is no need to reinvent the wheel, or add distractions such as
> infoboxes and other bells and whistles.
> 
> I find it extraordinary that, after 13 years, a project designed to make
> the sum of human knowledge available to humanity, with an annual budget of
> $50 million, has no clue how to measure the quality of the content it is
> providing, no apparent interest in doing so, and no apparent will to spend
> money on it.
> 
> For what it's worth, there was a recent external study of Wikipedia's
> medical content that came to unflattering results:
> 
> http://www.jaoa.org/content/114/5/368.full
> 
> ---o0o---
> 
> Most Wikipedia articles for the 10 costliest conditions in the United
> States contain errors compared with standard peer-reviewed sources. Health
> care professionals, trainees, and patients should use caution when using
> Wikipedia to answer questions regarding patient care.
> 
> Our findings reinforce the idea that physicians and medical students who
> currently use Wikipedia as a medical reference should be discouraged from
> doing so because of the potential for errors.
> 
> ---o0o---
> 
> 
> On Wed, May 7, 2014 at 10:59 PM, Risker  wrote:
> 
>> On 7 May 2014 16:17, Anthony Cole  wrote:
>> 
>>> Could someone please point me to all the studies the WMF have conducted
>>> into the reliability of Wikipedia's content? I'm particularly interested
>> in
>>> the medical content, but would also like to look over the others too.
>>> Cheers.
>>> 
>>> Anthony Cole 
>>> 


___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

Junk science? I suppose the Article Feedback Tool was more scientific,
then, because that's the best the Foundation has come up with so far.


On Wed, May 7, 2014 at 11:27 PM, Risker  wrote:

> On 7 May 2014 18:14, Andreas Kolbe  wrote:
>
> > Anne, there are really well-established systems of scholarly peer review.
> > There is no need to reinvent the wheel, or add distractions such as
> > infoboxes and other bells and whistles.
> >
> > I find it extraordinary that, after 13 years, a project designed to make
> > the sum of human knowledge available to humanity, with an annual budget
> of
> > $50 million, has no clue how to measure the quality of the content it is
> > providing, no apparent interest in doing so, and no apparent will to
> spend
> > money on it.
> >
> > For what it's worth, there was a recent external study of Wikipedia's
> > medical content that came to unflattering results:
> >
> > http://www.jaoa.org/content/114/5/368.full
> >
> > ---o0o---
> >
> > Most Wikipedia articles for the 10 costliest conditions in the United
> > States contain errors compared with standard peer-reviewed sources.
> Health
> > care professionals, trainees, and patients should use caution when using
> > Wikipedia to answer questions regarding patient care.
> >
> > Our findings reinforce the idea that physicians and medical students who
> > currently use Wikipedia as a medical reference should be discouraged from
> > doing so because of the potential for errors.
> >
> >
>
> > Doesn't help very much in assessing the quality of the article on
> > [[Liancourt Rocks]] - when depending on where in the world one is, the
> > article can be reasonably accurate or completely inaccurate.  This is one
> > of the geographic issues of which I speak.
>
>
> There are also issues with the study you reference - it's quite biased
> toward American information and the articles only have two reviewers. It
> perhaps points out how easy it is to get junk science published in
> peer-reviewed journals if the topic is "sexy" enough - their own
> study wouldn't meet our standards for inclusion.
>
> Risker/Anne
> ___
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

On Wed, May 7, 2014 at 11:32 PM, Michael Maggs  wrote:

> Measuring the quality of Wikipedia articles in general is an issue that
> Wikimedia UK is interested in looking at, though by means of automation
> rather than the gold-standard but much less scalable method of scholarly
> peer review.
>


It doesn't *have* to be scalable. That's what sampling was invented for.

Automation. As they say, if all you have is a hammer, every problem looks
like a nail.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Risker]

On 7 May 2014 18:30, Andreas Kolbe  wrote:

> "In a blinded process, we randomly selected 10 reviewers to examine 2 of
> the selected Wikipedia articles. Each reviewer was an internal medicine
> resident or rotating intern at the time of the assignment. This arrangement
> created redundancy, giving the study 2 independent reviewers for each
> article. Also, by using physicians as reviewers, we ensured a baseline
> competency in medical literature interpretation and research."
>
> The articles reviewed were coronary artery disease, lung cancer, major
> depressive disorder, osteoarthritis, chronic obstructive pulmonary disease,
> hypertension, diabetes mellitus, back pain, hyperlipidemia and concussion.
>
> Carry on.
>
>


Ah, but the costliest conditions aren't actually comparable to the relevant
Wikipedia articles.  For example, the "costly condition" of cancer is
compared to the article on lung cancer, despite the fact that we have an
article on cancer.  The costly condition of "trauma-related disorders" - a
very broad topic that would include traumatic amputations, fractures,
burns, and a multitude of other issues is compared to the article on
concussion;  the costly condition of "mental disorders" is compared to the
article on  major depressive disorder despite, again, haing an article on
mental disorders.

And each article is reviewed by only two people; when one looks at the
results, we see that in most cases the two reviewers provided very
different results.

Risker/Anne
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Michael Maggs]

It does have to be scalable if you want to be able to measure any article, at 
least approximately. Let's say I am interested in 5000 articles on the subjects 
X, Y and Z, none of which have been manually rated, and never will be due to 
the scaling problem.  An automated tool would be extremely useful to me, and is 
the best measure I am going to get in the absence of huge resources to measure 
their qualities in the manual way. 

Michael


On 7 May 2014, at 23:38, Andreas Kolbe  wrote:

> On Wed, May 7, 2014 at 11:32 PM, Michael Maggs  wrote:
> 
>> Measuring the quality of Wikipedia articles in general is an issue that
>> Wikimedia UK is interested in looking at, though by means of automation
>> rather than the gold-standard but much less scalable method of scholarly
>> peer review.
>> 
> 
> 
> It doesn't *have* to be scalable. That's what sampling was invented for.
> 
> Automation. As they say, if all you have is a hammer, every problem looks
> like a nail.
> ___
> Wikimedia-l mailing list, guidelines at: 
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 
> 


___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

On Wed, May 7, 2014 at 11:40 PM, Risker  wrote:

> Ah, but the costliest conditions aren't actually comparable to the relevant
> Wikipedia articles.  For example, the "costly condition" of cancer is
> compared to the article on lung cancer, despite the fact that we have an
> article on cancer.  The costly condition of "trauma-related disorders" - a
> very broad topic that would include traumatic amputations, fractures,
> burns, and a multitude of other issues is compared to the article on
> concussion;  the costly condition of "mental disorders" is compared to the
> article on  major depressive disorder despite, again, haing an article on
> mental disorders.
>

Yes, and hypertension was compared to the WP article on hypertension,
hyperlipidemia was compared to the WP article on hyperlipidemia, etc.

And each article is reviewed by only two people; when one looks at the
> results, we see that in most cases the two reviewers provided very
> different results.
>

Frankly, the merits of this study are neither here not there. I just
thought it might be of interest.

What troubles me more is that you seem to be saying that meaningful expert
review of Wikipedia content is not possible; at least you have not given
any indication so far that you think otherwise. Yet beyond the confines of
Wikipedia, expert review happens routinely, and daily, and is widely relied
upon.

Is it really your belief that academics have no better access to knowledge
than does an undisciplined crowd of random people, and that there is no way
one could design a study that would give meaningful insight into the
current quality level of Wikipedia content? I would find that truly bizarre.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[phoebe ayers]

On Wed, May 7, 2014 at 1:17 PM, Anthony Cole  wrote:

> Could someone please point me to all the studies the WMF have conducted
> into the reliability of Wikipedia's content? I'm particularly interested in
> the medical content, but would also like to look over the others too.
> Cheers.
>
> Anthony Cole 
> ___
>

Hi Anthony,

There have been a number of studies by researchers looking at various
subsets of Wikipedia's medical articles; do you know about the list here?
https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Medicine/Wikipedia_and_medicine


The research group at the WMF to the best of my knowledge hasn't run any
reliability-of-articles studies itself, but there have been lots done by
domain experts in various fields.

best,
Phoebe


-- 
* I use this address for lists; send personal messages to phoebe.ayers 
gmail.com *
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Erik Moeller]

On Wed, May 7, 2014 at 4:12 PM, phoebe ayers  wrote:
> The research group at the WMF to the best of my knowledge hasn't run any
> reliability-of-articles studies itself, but there have been lots done by
> domain experts in various fields.

We commissioned one:
https://meta.wikimedia.org/wiki/Research:Accuracy_and_quality_of_Wikipedia_entries

We also did this experiment a while ago, which is IMO the best model
for a generalizable approach (i.e. work with groups of credentialed
experts, provide a simple API):

http://blog.wikimedia.org/2010/12/09/encyclopedia-of-life-curates-wikipedias-species-articles/

I don't think it should be our highest priority, but I do think more
research and development in this area is warranted.

Erik
-- 
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[phoebe ayers]

On Wed, May 7, 2014 at 3:14 PM, Andreas Kolbe  wrote:

> Anne, there are really well-established systems of scholarly peer review.
> There is no need to reinvent the wheel, or add distractions such as
> infoboxes and other bells and whistles.


And those peer review systems have lots and lots of problems as well as
upsides. Lots of people *are* trying to reinvent peer review, including
some very respected scientists.* As an academic science librarian, I can
attest to there being widespread and currently ongoing debates about how to
review scientific knowledge, whether traditional peer review is sufficient,
and how to improve it. The current system for scientific research is often
opaque, messy, prone to failure and doesn't always support innovation, and
lots of smart people are thinking about it.

Erik: aha! I'd forgotten about those case studies, thanks!

-- phoebe

* http://blogs.berkeley.edu/2013/10/04/open-access-is-not-the-problem/
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

On Thu, May 8, 2014 at 12:22 AM, phoebe ayers  wrote:

> On Wed, May 7, 2014 at 3:14 PM, Andreas Kolbe  wrote:
>
> > Anne, there are really well-established systems of scholarly peer review.
> > There is no need to reinvent the wheel, or add distractions such as
> > infoboxes and other bells and whistles.
>
>
> And those peer review systems have lots and lots of problems as well as
> upsides. Lots of people *are* trying to reinvent peer review, including
> some very respected scientists.* As an academic science librarian, I can
> attest to there being widespread and currently ongoing debates about how to
> review scientific knowledge, whether traditional peer review is sufficient,
> and how to improve it. The current system for scientific research is often
> opaque, messy, prone to failure and doesn't always support innovation, and
> lots of smart people are thinking about it.
>
> Erik: aha! I'd forgotten about those case studies, thanks!
>

Given that the post that started this thread referenced medical content,
are you telling me that you think it would be useless to have qualified
medical experts reviewing Wikipedia's medical content, because the process
would be "opaque, messy, prone to failure and doesn't always support
innovation"?
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Thyge]

mr Andreas Kolbe,
I would like to tell you, that your mailings here strike me as being
negative and unhelpful.
If you have any suggestions for improvement, please put them forward, since
this is an interesting topic.
The "undisciplined crowd of random people" is what the world comprises, and
a subset of those are trying their best to bring knowledge to the world and
appreciate any help you may provide to improve and measure quality.

regards,
Thyge


2014-05-08 1:38 GMT+02:00 Andreas Kolbe wrote
>

Given that the post that started this thread referenced medical content,
> are you telling me that you think it would be useless to have qualified
> medical experts reviewing Wikipedia's medical content, because the process
> would be "opaque, messy, prone to failure and doesn't always support
> innovation"?
> ___
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

On Thu, May 8, 2014 at 12:48 AM, Thyge  wrote:

> mr Andreas Kolbe,
> I would like to tell you, that your mailings here strike me as being
> negative and unhelpful.
>

:)


> If you have any suggestions for improvement, please put them forward, since
> this is an interesting topic.
> The "undisciplined crowd of random people" is what the world comprises,


That's not all the world comprises. There are universities.


> and
> a subset of those are trying their best to bring knowledge to the world and
> appreciate any help you may provide to improve and measure quality.
>

As for study design, I'd suggest you begin with a *random* sample of
frequently-viewed Wikipedia articles in a given topic area (e.g. those
within the purview of WikiProject Medicine), have them assessed by an
independent panel of academic experts, and let them publish their results.

All of that is quite doable. You begin with a list of articles from the
database, agree a method of random selection, and let experts do their job.

If the results are good, it redounds to Wikipedia's credit. If the results
are bad, it provides valuable feedback to the community, an indication of
Wikipedia's reliability to the public, an opportunity for further analysis
both within and without the Wikipedia community, and an indication of where
quality improvement efforts should be focused.

These are all outcomes that are fully in line with the Foundation's mission.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Risker]

On 7 May 2014 19:38, Andreas Kolbe  wrote:

> On Thu, May 8, 2014 at 12:22 AM, phoebe ayers 
> wrote:
>
> > On Wed, May 7, 2014 at 3:14 PM, Andreas Kolbe 
> wrote:
> >
> > > Anne, there are really well-established systems of scholarly peer
> review.
> > > There is no need to reinvent the wheel, or add distractions such as
> > > infoboxes and other bells and whistles.
> >
> >
> > And those peer review systems have lots and lots of problems as well as
> > upsides. Lots of people *are* trying to reinvent peer review, including
> > some very respected scientists.* As an academic science librarian, I can
> > attest to there being widespread and currently ongoing debates about how
> to
> > review scientific knowledge, whether traditional peer review is
> sufficient,
> > and how to improve it. The current system for scientific research is
> often
> > opaque, messy, prone to failure and doesn't always support innovation,
> and
> > lots of smart people are thinking about it.
> >
> > Erik: aha! I'd forgotten about those case studies, thanks!
> >
>
> Given that the post that started this thread referenced medical content,
> are you telling me that you think it would be useless to have qualified
> medical experts reviewing Wikipedia's medical content, because the process
> would be "opaque, messy, prone to failure and doesn't always support
> innovation"?
>


Andreas, I don't think that's necessarily what is being said here.
However, the review needs to be scientifically valid, and the review in the
JAOA isn't.  For example, it does not require that the assessor look at the
references used in the article to determine whether or not the reference
meets the arbitratory standard set (i.e. peer-reviewed source updated or
published within the last 5 years), and whether or not the article says
what the reference says.  Instead, the assessors looked at sources that may
or may not have been used in the article, thus eroding any control for
disagreement amongst scientific peers - something that most editors who
work in this area know is surprisingly common.

The study itself identifies very significant, possibly fatal, limitations,
including the use of essentially random reference sources that just happen
to be available, the level of understanding of the subjects by the
reviewers, the limited number of reviewers, and the fact that subject
matter experts themselves are often in disagreement.  It has not
demonstrated repeatability.

It's possible to create a study that's worthwhile.  This one wasn't it.

Risker/Anne
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Thyge]

Maybe you should suggest that to the universities and not just to this
mailing list.
Nothing prevents to set up " an independent panel of academic experts" and
to start doing that job today.
regards,
Thyge



2014-05-08 2:00 GMT+02:00 Andreas Kolbe :

> On Thu, May 8, 2014 at 12:48 AM, Thyge  wrote:
>
> > mr Andreas Kolbe,
> > I would like to tell you, that your mailings here strike me as being
> > negative and unhelpful.
> >
>
> :)
>
>
> > If you have any suggestions for improvement, please put them forward,
> since
> > this is an interesting topic.
> > The "undisciplined crowd of random people" is what the world comprises,
>
>
> That's not all the world comprises. There are universities.
>
>
> > and
> > a subset of those are trying their best to bring knowledge to the world
> and
> > appreciate any help you may provide to improve and measure quality.
> >
>
> As for study design, I'd suggest you begin with a *random* sample of
> frequently-viewed Wikipedia articles in a given topic area (e.g. those
> within the purview of WikiProject Medicine), have them assessed by an
> independent panel of academic experts, and let them publish their results.
>
> All of that is quite doable. You begin with a list of articles from the
> database, agree a method of random selection, and let experts do their job.
>
> If the results are good, it redounds to Wikipedia's credit. If the results
> are bad, it provides valuable feedback to the community, an indication of
> Wikipedia's reliability to the public, an opportunity for further analysis
> both within and without the Wikipedia community, and an indication of where
> quality improvement efforts should be focused.
>
> These are all outcomes that are fully in line with the Foundation's
> mission.
> ___
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

On Thu, May 8, 2014 at 1:06 AM, Thyge  wrote:

> Maybe you should suggest that to the universities and not just to this
> mailing list.
> Nothing prevents to set up " an independent panel of academic experts" and
> to start doing that job today.
> regards,
> Thyge
>


Well, I'd like the Foundation to invest in such research, which is why I
brought it up here.

I cant think of several instances of donors' money being spent on things
that to me seemed less supportive of the Foundation's core mission.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Nathan]

On Wed, May 7, 2014 at 8:12 PM, Andreas Kolbe  wrote:
>
>
>
> Well, I'd like the Foundation to invest in such research, which is why I
> brought it up here.
>
> I cant think of several instances of donors' money being spent on things
> that to me seemed less supportive of the Foundation's core mission.
> ___
>

Perhaps while the UK chapter pursues automated methods of assessment,
another chapter can apply for a WMF grant to pursue a more traditional
review effort. Maybe Wikimedia DC? I don't think this kind of research is
really the WMF's purview; for reasons everyone is familiar with, it's
important they remain distant from reviewing and managing content.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Wil Sinclair]

I'm a total newb here, and I know the grant system between WMF and the
different chapters has been debated in the past. But I have a simple
question: if WMF is funding these efforts through grants and the grant
money is used to review and/or manage content, wouldn't it be
indirectly getting involved with reviewing and managing content?

,Wil

On Wed, May 7, 2014 at 5:17 PM, Nathan  wrote:
> On Wed, May 7, 2014 at 8:12 PM, Andreas Kolbe  wrote:
>>
>>
>>
>> Well, I'd like the Foundation to invest in such research, which is why I
>> brought it up here.
>>
>> I cant think of several instances of donors' money being spent on things
>> that to me seemed less supportive of the Foundation's core mission.
>> ___
>>
>
> Perhaps while the UK chapter pursues automated methods of assessment,
> another chapter can apply for a WMF grant to pursue a more traditional
> review effort. Maybe Wikimedia DC? I don't think this kind of research is
> really the WMF's purview; for reasons everyone is familiar with, it's
> important they remain distant from reviewing and managing content.
> ___
> Wikimedia-l mailing list, guidelines at: 
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 
> 

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Nathan]

On Wed, May 7, 2014 at 8:24 PM, Wil Sinclair  wrote:

> I'm a total newb here, and I know the grant system between WMF and the
> different chapters has been debated in the past. But I have a simple
> question: if WMF is funding these efforts through grants and the grant
> money is used to review and/or manage content, wouldn't it be
> indirectly getting involved with reviewing and managing content?
>
> ,Wil
>
>
Depends on the nature of the grant. In any case I think affiliates are
better placed to perform this kind of work anyway, since we'd want it to be
done in more than one language and using diverse panels with members from
more than just the U.S. But I do think it would be really cool research and
the results would certainly be very interesting. It also makes sense as
complementary to automated efforts, and then the results of the different
methods could be compared to assess effectiveness of the review processes.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Wil Sinclair]

I looked at WMF's grant page here:
http://meta.wikimedia.org/wiki/Grants. I don't see any mention of
grants for academic research. Does the WMF give such grants? If not,
why not?

,Wil

On Wed, May 7, 2014 at 5:12 PM, Andreas Kolbe  wrote:
> On Thu, May 8, 2014 at 1:06 AM, Thyge  wrote:
>
>> Maybe you should suggest that to the universities and not just to this
>> mailing list.
>> Nothing prevents to set up " an independent panel of academic experts" and
>> to start doing that job today.
>> regards,
>> Thyge
>>
>
>
> Well, I'd like the Foundation to invest in such research, which is why I
> brought it up here.
>
> I cant think of several instances of donors' money being spent on things
> that to me seemed less supportive of the Foundation's core mission.
> ___
> Wikimedia-l mailing list, guidelines at: 
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 
> 

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Nathan]

On Wed, May 7, 2014 at 8:45 PM, Wil Sinclair  wrote:

> I looked at WMF's grant page here:
> http://meta.wikimedia.org/wiki/Grants. I don't see any mention of
> grants for academic research. Does the WMF give such grants? If not,
> why not?
>
> ,Wil



https://meta.wikimedia.org/wiki/Grants:IEG/The_use_of_Wikipedia_by_doctors_for_their_information_needs

The WMF focuses on submitted grant proposals. The IEG grants cover a pretty
wide range of projects, but some of them are research oriented and involve
academics and academic institutions. While WMF executives can make a much
better judgment of the boundaries of content involvement, there seems to be
a substantial difference between agreeing to fund grants and commissioning
specific research into a core topic like quality. However, affiliates
should have no such concerns. I really hope someone picks up on this kind
of project, perhaps I'll suggest it on the Wikimedia DC list or to
Wikimedia New England.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

On Thu, May 8, 2014 at 1:30 AM, Nathan  wrote:

> On Wed, May 7, 2014 at 8:24 PM, Wil Sinclair  wrote:
>
> > I'm a total newb here, and I know the grant system between WMF and the
> > different chapters has been debated in the past. But I have a simple
> > question: if WMF is funding these efforts through grants and the grant
> > money is used to review and/or manage content, wouldn't it be
> > indirectly getting involved with reviewing and managing content?
> >
> > ,Wil
> >
> >
> Depends on the nature of the grant. In any case I think affiliates are
> better placed to perform this kind of work anyway, since we'd want it to be
> done in more than one language and using diverse panels with members from
> more than just the U.S. But I do think it would be really cool research and
> the results would certainly be very interesting. It also makes sense as
> complementary to automated efforts, and then the results of the different
> methods could be compared to assess effectiveness of the review processes.
>


I don't think this is an issue; as Erik has kindly pointed out in this
thread, the Foundation has funded at least one such study in the past.
(However, this study does not seem to have been based on a random sample –
at least I cannot find any mention of the sample selection method in the
study's write-up. The selection of a random sample is key to any such
effort, and the method used to select the sample should be described in
detail in any resulting report.)

To me, funding work that results in content quality feedback to the
community does not mean that the Foundation is getting involved in content
management. The expert panel would obviously have to have complete academic
freedom to publish whatever their findings are, without pre-publication
review by the Foundation. I would not expect the experts involved to end up
editing Wikipedia; if any of them did, this would be their private
initiative as individuals, and not covered by any grant.

I would consider such a research programme an important service to the
community, just as the Board provides software, guidance through board
resolutions, and so forth.

It would be an equally vital service to the reading public that the
Foundation's projects serve.

In my view, any such programme of studies should begin with the English
Wikipedia, as it is the most comprehensive and most widely accessed
project, including by many non-native speakers looking for more detailed
information than their own language version of Wikipedia provides. Medical
content would be an excellent area to start with.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

[Wikimedia-l] Amical Wikimedia Report, April 2014

[David Parreño Mont - Comunicació]

Dear fellows,

The following message is just to keep you informed about the activities
developed in April by Amical Wikimedia.

https://meta.wikimedia.org/wiki/Amical_Wikimedia/April_2014

Kind regards,

David Parreño Mont
User:Davidpar
Communications, Amical Wikimedia



*Source:* https://meta.wikimedia.org/wiki/Amical_Wikimedia/April_2014

GLAM-WIKI10 hours edit-a-thon for 100 design students

During April we did the central activity of the project where we are mixing
GLAM and EDU with the Design Museum of Barcelona. Around 400 students in 25
classrooms of 13 design schools are improving Wikipedia Articles. On April
25 we did a 10 hours long design editathon (organized in groups from 10 to
20 people) at the Documentation Center of the Museum. Click here[2] to see
more pics. It was the opening day of the documentation center so students
could behave as beta testers. One of the good points was doing it
graduatelly with small groups so we could help them personally and also
that they could take the books back home (15 days free-rent), so Wikipedia
articles are being improved since then. We also did a backstage pass to the
warehouse of the documentation center and a general guided tour to the
building.
Outreach

   - April 3rd- Conference: *Projectes voluntaris i col·laboratius de
   traducció* (Volunteer driven translation projects) at Col·legi de
   Doctors i Llicenciats (url): 25 attendees - Kippelboy
   - 28th January to 14th April — Il·lustraciència Award[3] from Associació
   Catalana de Comunicació Científica (Catalan Asociation of Science
   Comunication) give to the participants the option to accept Wikimedia
   Commons upload of their artworks. Post in the blog 12th february[4].
   Results: 211 artworks liberated[5] (upload soon)
   - April 10th - Public presentation of Wikiproject Barcelona's popular
   culture[6] (done in collaboration with Town council)
   - April 25th - 10 hours design editathon at the Museu del Disseny de
   Barcelona. (100 attendees)

Projects

   - 1st April - Joan Brossa Foundation releases wikiquote material[7] of
   the artist and help us referencing existing one.
   - 28th April - "Colita: la reivindicació de la mirada"[8] conference
   press pass for a Wikimedia Commons contributor. Judesba
   - Ongoing project: Wikiproject Public Libraries
   - Ongoing project: Wikiproject Miró
   - Ongoing project: Víctor Balaguer Museum project
   - Ongoing project: Music Museum in Barcelona
   - Ongoing project: Museu Morera in Lleida
   - Ongoing project: CCCB, Contemporary cultural center in Barcelona
   - Some other smaller GLAMWIki projects

Meetings

   - April 3rd - Scientific Committee meeting for organizing "Libraries and
   heritage conference" at Ateneu Barcelonès in Nov 20. Kippelboy
   - April 4th - GLAMwiki projects planning meeting with Museu Blau-Natural
   History Museum in Barcelona. Vàngelis Villar
   - April 22d - GLAMwiki projects planning meeting with National
   Archaeology Museum in Tarragona- Kippelboy & Lluis_tgn
   - April 22d - GLAMwiki projects planning meeting with local
*Hemerotèque*in Tarragona- Kippelboy & Lluis_tgn
   - April 22d - GLAMwiki follow up meeting with Victor Balaguer Museum-
   Kippelboy
   - April 28th - Meeting with Lavinia Productora to talk about a project
   proposal- Kippelboy
   - April 29th - Follow up Meeting with Head of Catalan Public Libraries
   Network- Kippelboy
   - April 30th - Introduction GLAMWIKI Meeting with Sitges Museums
   Director- Kippelboy

Education1s Wikipedian in Residence in a EU University.

User Jey is the brand new Wikipedian in Residence of the Open University of
Catalonia[9]. Project has been designed in phases. On this phase 1 his main
goals will be mapping the already published Open Materials in the universty
(CC-BY-SA docs, Opencoursewares,...), specially focusing on Audiovisual
materials (videos). Also mapping the teachers who are already doing
Wikipedia-related projects and trying to scale this opportunities to the
rest of the proffessors. The first phase of the residency will last from
April to July 2014.
Outreach

   - April 8th - Wikipedia Editing course at La Bisbal d'Empordà Highschool
   - Anskar
   - April 24th - Kick of meeting for the First ever Wikipedian in
   Residence in an European University[10] (UOC) - Kippelboy Jey
   - April 28th - Wikipedia presentation to 35 teachers of the Barceloneta
   high-school, organized by Barceloneta library - Kippelboy
   - April 28th - Wikipedia presentation to 80 schoolars (9 & 10 y.o.) of
   la Garriga (els Pinetons school) - Amadalvarez
   - April 29th - Amical's representation at the awards ceremony of the
   websalpuntcat award (Teenage Catalan language websites contest) - Kippelboy
   - wikiArS
  - April 1th to 30th. Exhibition wikiArS: contribució de les escoles
  d'art i disseny a Viquipèdia[11] (*wikiArS: contributions from art &
  design schools to Wikipedia*) in the Badalona central Library Can
  Casacuberta.
  - Apr

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Risker]

On 7 May 2014 20:56, Andreas Kolbe  wrote:

> On Thu, May 8, 2014 at 1:30 AM, Nathan  wrote:
>
> > On Wed, May 7, 2014 at 8:24 PM, Wil Sinclair  wrote:
> >
> > > I'm a total newb here, and I know the grant system between WMF and the
> > > different chapters has been debated in the past. But I have a simple
> > > question: if WMF is funding these efforts through grants and the grant
> > > money is used to review and/or manage content, wouldn't it be
> > > indirectly getting involved with reviewing and managing content?
> > >
> > > ,Wil
> > >
> > >
> > Depends on the nature of the grant. In any case I think affiliates are
> > better placed to perform this kind of work anyway, since we'd want it to
> be
> > done in more than one language and using diverse panels with members from
> > more than just the U.S. But I do think it would be really cool research
> and
> > the results would certainly be very interesting. It also makes sense as
> > complementary to automated efforts, and then the results of the different
> > methods could be compared to assess effectiveness of the review
> processes.
> >
>
>
> I don't think this is an issue; as Erik has kindly pointed out in this
> thread, the Foundation has funded at least one such study in the past.
> (However, this study does not seem to have been based on a random sample –
> at least I cannot find any mention of the sample selection method in the
> study's write-up. The selection of a random sample is key to any such
> effort, and the method used to select the sample should be described in
> detail in any resulting report.)
>
> To me, funding work that results in content quality feedback to the
> community does not mean that the Foundation is getting involved in content
> management. The expert panel would obviously have to have complete academic
> freedom to publish whatever their findings are, without pre-publication
> review by the Foundation. I would not expect the experts involved to end up
> editing Wikipedia; if any of them did, this would be their private
> initiative as individuals, and not covered by any grant.
>
> I would consider such a research programme an important service to the
> community, just as the Board provides software, guidance through board
> resolutions, and so forth.
>
> It would be an equally vital service to the reading public that the
> Foundation's projects serve.
>
> In my view, any such programme of studies should begin with the English
> Wikipedia, as it is the most comprehensive and most widely accessed
> project, including by many non-native speakers looking for more detailed
> information than their own language version of Wikipedia provides. Medical
> content would be an excellent area to start with.
>


I think perhaps there is a lack of research into the extent of research
already being done by independent, qualified third parties.  Several
examples are provided in the references of the study you posted, Andreas.
For example, this one in the Journal of Oncology Practice[1] compares
specific Wikipedia articles for patient-oriented cancer information against
the professionally edited PDQ database.  It appears that the two were
comparable in most areas, except for readability, where the PDQ database
was considered significantly more readable.  Now, again, this is a small
study and it has not been reproduced; however, it took me minutes to find
more information on the very subject you're interested in, created by
completely independent bodies who have "no pony in the race". There did
seem to be a fair number of studies related to medical topics.  Now if only
we could learn from them - especially on the readability point, which I
think really is a very serious issue.  Wikipedia isn't really intended to
educate physicians about medical topics, it's intended to be a general
reference for non-specialists.

Very few people are going to make life-and-death decisions based on our
math or physics topic areas, but I'll lay odds that any study would find a
significant readability issue with both of them, as well.

Risker/Anne

[1]  http://jop.ascopubs.org/content/7/5/319.full
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Wil Sinclair]

Would it be possible for WMF or another organization to initiate and
potentially fund a project modeled on the Human Genome Project? That
is, WMF or some other institution could host a large database of data
that researchers can contribute to and that makes all the data
available for researchers to analyze and build visualization tools
against? Kinda like a wiki based on data from analyzing another wiki.
;) Such data might add a set of metadata on an article that could be
used as a field in a hypercube, for example.

Beyond just hosting the database, it would be possible to write tools
that check aspects of these data before commit to make sure that they
are consistent with the other data that has already been committed.
Large data set leave very unique signatures in aggregate. For example,
we use checksums all the time to verify that data has been corrupted
during transmissions.

I imagine this isn't the first time someone has thrown something like
this in to the Wikipedosphere. If so, what did people think? If not,
what do you guys think? :)

,Wil

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Nathan]

On Wed, May 7, 2014 at 10:10 PM, Wil Sinclair  wrote:
>
>
> I imagine this isn't the first time someone has thrown something like
> this in to the Wikipedosphere. If so, what did people think? If not,
> what do you guys think? :)
>
> ,Wil
>
>
I think it sounds a little bit like wikidata.org, with some innovation of
potential future applications.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

On Thu, May 8, 2014 at 2:41 AM, Risker  wrote:
>
>
> I think perhaps there is a lack of research into the extent of research
> already being done by independent, qualified third parties.  Several
> examples are provided in the references of the study you posted, Andreas.
> For example, this one in the Journal of Oncology Practice[1] compares
> specific Wikipedia articles for patient-oriented cancer information against
> the professionally edited PDQ database.  It appears that the two were
> comparable in most areas, except for readability, where the PDQ database
> was considered significantly more readable.  Now, again, this is a small
> study and it has not been reproduced; however, it took me minutes to find
> more information on the very subject you're interested in, created by
> completely independent bodies who have "no pony in the race". There did
> seem to be a fair number of studies related to medical topics.  Now if only
> we could learn from them - especially on the readability point, which I
> think really is a very serious issue.  Wikipedia isn't really intended to
> educate physicians about medical topics, it's intended to be a general
> reference for non-specialists.
>
> Very few people are going to make life-and-death decisions based on our
> math or physics topic areas, but I'll lay odds that any study would find a
> significant readability issue with both of them, as well.
>
> Risker/Anne
>
> [1]  http://jop.ascopubs.org/content/7/5/319.full
>
>

In the study you reference, Anne, reviewers spent all of 18 minutes on each
article. The readability analysis was done by automation.

I did review the link Phoebe posted earlier;

https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Medicine/Wikipedia_and_medicine

I found the number of recent studies assessing actual Wikipedia content in
this field there very scanty. The best seemed to be this February 2014
study in the European Journal of Gastroenterology & Hepatology:

http://www.ncbi.nlm.nih.gov/pubmed/24276492

The conclusion of that study, based on a review of 39 articles by three
assessors, was that

—o0o—

"Wikipedia is not a reliable source of information for medical students
searching for gastroenterology and hepatology articles. Several
limitations, deficiencies, and scientific errors have been identified in
the articles examined."

—o0o—

There was also this study, concluding that Wikipedia was "fairly reliable":

http://onlinelibrary.wiley.com/doi/10./sdi.12059/full

But they say their reliability assessment was based on a simple count of
references, without reviewing the actual article content for accuracy!

—o0o—

Assessment of Reliability

The reliability of nephrology articles in Wikipedia was determined in two
ways: (i) mean number of references per article, and (ii) mean percentage
of “substantiated” references—which we defined as references corresponding
to works published in peer-reviewed journals or from texts with an
associated International Standard Book Number (ISBN).
—o0o—

As far as I am aware from discussions with members of Wiki Project Med
Foundation/WikiProject Medicine, all the studies that have been done to
date suffer from small sample size or other methodological limitations.
There is a real gap for a large-scale, well-designed study of a random
subset of Wikipedia's most-consulted medical articles.*

If Wikipedia wants to be serious about its mission, measuring article
quality is a must. Given how widely Wikipedia is used today, it is also a
question of social responsibility.

I imagine the reluctance I sense in this discussion is in part due to
people's fear that the results might be less than stellar. If so, that fear
is misplaced. There is no improvement without performance feedback. If the
results are indeed disappointing, the related publicity should lead to an
increased focus on improvement efforts, and indeed may encourage a greater
influx of better-qualified editors if such are needed.

The project is well into its second decade. It is mature and
well-established enough for such a test. It's a question of taking a
long-term view. Long-term improvement will be accelerated by honest,
knowledgeable feedback.

Studies could be repeated at annual intervals, to track progress and
measure improvement. I don't believe there is any other way of arriving at
a reliable reference source, which after all is what this entire effort
should be about.


* Some ongoing related discussions at WikiProject Medicine here:

https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Medicine#This_conversation_is_notable
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Risker]

On 7 May 2014 22:24, Andreas Kolbe  wrote:

> On Thu, May 8, 2014 at 2:41 AM, Risker  wrote:
> >
> >
> > I think perhaps there is a lack of research into the extent of research
> > already being done by independent, qualified third parties.  Several
> > examples are provided in the references of the study you posted, Andreas.
> > For example, this one in the Journal of Oncology Practice[1] compares
> > specific Wikipedia articles for patient-oriented cancer information
> against
> > the professionally edited PDQ database.  It appears that the two were
> > comparable in most areas, except for readability, where the PDQ database
> > was considered significantly more readable.  Now, again, this is a small
> > study and it has not been reproduced; however, it took me minutes to find
> > more information on the very subject you're interested in, created by
> > completely independent bodies who have "no pony in the race". There did
> > seem to be a fair number of studies related to medical topics.  Now if
> only
> > we could learn from them - especially on the readability point, which I
> > think really is a very serious issue.  Wikipedia isn't really intended to
> > educate physicians about medical topics, it's intended to be a general
> > reference for non-specialists.
> >
> > Very few people are going to make life-and-death decisions based on our
> > math or physics topic areas, but I'll lay odds that any study would find
> a
> > significant readability issue with both of them, as well.
> >
> > Risker/Anne
> >
> > [1]  http://jop.ascopubs.org/content/7/5/319.full
> >
> >
>
> In the study you reference, Anne, reviewers spent all of 18 minutes on each
> article. The readability analysis was done by automation.
>
>

Yes, of course readability analysis is done by automation.  I've yet to
find a consistent readability assessment that doesn't use automation.  It's
not an area where subjectivity is particularly useful.

And that was an average of 18 minutes per article, i.e., 36 minutes: 18
minutes for the WP article and 18 minutes for the PDQ article.  How long do
you really think it should take?  I read several of the articles in under 5
minutes on each site.  Of course, the reviewers wouldn't need to look up
the definitions of a lot of the terms that lay people would need to look
at, because they were already professionally educated in the topic area, so
that would significantly reduce the amount of time required to assess the
article.

Andreas, you seem to have pre-determined that Wikipedia's medical articles
are all terrible and riddled with errors.  Realistically, they're amongst
the most likely to receive professional editing and review - Wikiproject
Medicine does a much better job than people are willing to credit them.
The biggest weakness to the articles - and I've heard this from many people
who read them - is that they're written at too high a level to be really
accessible to lay people.  I thought the point that the study made about
the benefit of linking to an "English" dictionary definition of complex
terms rather than to another highly technical Wikipedia article was a very
good one, for example.  We could learn from these studies.

Indeed, many science articles are mainly written by professionals in the
field (I noted math and physics earlier, but chemistry and of course a
large number of computer articles are also written by professionals).  The
biggest challenge for these subjects is to write them in an accessible way.
Note, I said "science" - alternative medicine, history, geopolitical and
"soft science" articles are much more problematic.

Risker/Anne
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Andreas Kolbe]

On Thu, May 8, 2014 at 3:41 AM, Risker  wrote:

> Yes, of course readability analysis is done by automation.  I've yet to
> find a consistent readability assessment that doesn't use automation.  It's
> not an area where subjectivity is particularly useful.
>
> And that was an average of 18 minutes per article, i.e., 36 minutes: 18
> minutes for the WP article and 18 minutes for the PDQ article.  How long do
> you really think it should take?  I read several of the articles in under 5
> minutes on each site.  Of course, the reviewers wouldn't need to look up
> the definitions of a lot of the terms that lay people would need to look
> at, because they were already professionally educated in the topic area, so
> that would significantly reduce the amount of time required to assess the
> article.
>


It took me more than 18 minutes to write the last e-mail in this thread. :)

The lung cancer article, for example, which was among those reviewed, has
well over 4,500 words of prose, and cites 141 references. That's a
reviewing speed of 250 words per minute. I don't know if you have ever done
an FA review ...



> Andreas, you seem to have pre-determined that Wikipedia's medical articles
> are all terrible and riddled with errors.



And I think you are being needlessly defensive. I have an open mind as to
what the results might be. What I am sure of is that neither you nor I nor
the Foundation really know how reliable they are. Why not make an effort to
find out?



> Realistically, they're amongst
> the most likely to receive professional editing and review - Wikiproject
> Medicine does a much better job than people are willing to credit them.
>


Yes, and many editors there are sorely concerned about the quality of
medical information Wikipedia provides to the public.

Incidentally, there was a discussion of the JAOA study in The Atlantic
today:

http://www.theatlantic.com/health/archive/2014/05/can-wikipedia-ever-be-a-definitive-medical-text/361822/

A member of WikiProject Medicine is quoted in it, as is the study's author.

—o0o—

So both sides acknowledge: There are errors in Wikipedia’s health articles.
And that’s a problem, because people use them.

—o0o—





> The biggest weakness to the articles - and I've heard this from many people
> who read them - is that they're written at too high a level to be really
> accessible to lay people.  I thought the point that the study made about
> the benefit of linking to an "English" dictionary definition of complex
> terms rather than to another highly technical Wikipedia article was a very
> good one, for example.  We could learn from these studies.
>
> Indeed, many science articles are mainly written by professionals in the
> field (I noted math and physics earlier, but chemistry and of course a
> large number of computer articles are also written by professionals).  The
> biggest challenge for these subjects is to write them in an accessible way.
> Note, I said "science" - alternative medicine, history, geopolitical and
> "soft science" articles are much more problematic.
>
> Risker/Anne
> ___
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Anthony Cole]

Wow.

Wil - you're going to love WikiData.

Phoebe: I  have seen that list of peer-reviewed articles related to
Wikipedia medical content. I've extracted those related to quality and
added more from a couple of database searches I did in January and the list
of 42 (some are letters and there's a conference abstract, though) are
collapsed on the WikiProject Medicine talk page now under the heading,
"This thread is notable."

I've read most but not all of those and, as Andreas mentioned, most of
those suffered from small sample size and poor or opaque sample selection
criteria.

Erik, thank you for pointing to the "reviewer" trial. I had read it before
and I'm glad to have this opportunity to tell you how much I love it. There
is a big hole in Wikipedia where expert reviewing belongs.

I'm presently on the board of WikiProject Med Foundation, but will be
stepping down after Wikimania. I mostly edit medical content. Anne is
right, it is heavily curated. But stuff slips through the net of patrollers
from time to time, and barely a day goes by without some howler of a
long-term problem coming to light.

I would like to know - know, rather than rely on my gut feeling - how
accurate our medical content is. To know that, I think the first step would
be to get an expert on scientific study design to review the 30-40 existing
studies that address the quality of our medical content, and tell us what,
if anything, we can take from that prior work - essentially what Anne
recommends above, but rather than making my own incompetent and heavily
biased assessment, get an expert to do it.

My own, inexpert, belief is that those studies are (mostly) so hopelessly
flawed that nothing can seriously be generalised from them. If I'm right,
I'd then like us all to consider seriously doing a survey whose design is
sufficiently rigorous to give us an answer.

Thanks for your thoughts and attention everyone.

Anthony Cole 



On Thu, May 8, 2014 at 11:05 AM, Andreas Kolbe  wrote:

> On Thu, May 8, 2014 at 3:41 AM, Risker  wrote:
>
> > Yes, of course readability analysis is done by automation.  I've yet to
> > find a consistent readability assessment that doesn't use automation.
>  It's
> > not an area where subjectivity is particularly useful.
> >
> > And that was an average of 18 minutes per article, i.e., 36 minutes: 18
> > minutes for the WP article and 18 minutes for the PDQ article.  How long
> do
> > you really think it should take?  I read several of the articles in
> under 5
> > minutes on each site.  Of course, the reviewers wouldn't need to look up
> > the definitions of a lot of the terms that lay people would need to look
> > at, because they were already professionally educated in the topic area,
> so
> > that would significantly reduce the amount of time required to assess the
> > article.
> >
>
>
> It took me more than 18 minutes to write the last e-mail in this thread. :)
>
> The lung cancer article, for example, which was among those reviewed, has
> well over 4,500 words of prose, and cites 141 references. That's a
> reviewing speed of 250 words per minute. I don't know if you have ever done
> an FA review ...
>
>
>
> > Andreas, you seem to have pre-determined that Wikipedia's medical
> articles
> > are all terrible and riddled with errors.
>
>
>
> And I think you are being needlessly defensive. I have an open mind as to
> what the results might be. What I am sure of is that neither you nor I nor
> the Foundation really know how reliable they are. Why not make an effort to
> find out?
>
>
>
> > Realistically, they're amongst
> > the most likely to receive professional editing and review - Wikiproject
> > Medicine does a much better job than people are willing to credit them.
> >
>
>
> Yes, and many editors there are sorely concerned about the quality of
> medical information Wikipedia provides to the public.
>
> Incidentally, there was a discussion of the JAOA study in The Atlantic
> today:
>
>
> http://www.theatlantic.com/health/archive/2014/05/can-wikipedia-ever-be-a-definitive-medical-text/361822/
>
> A member of WikiProject Medicine is quoted in it, as is the study's author.
>
> —o0o—
>
> So both sides acknowledge: There are errors in Wikipedia’s health articles.
> And that’s a problem, because people use them.
>
> —o0o—
>
>
>
>
>
> > The biggest weakness to the articles - and I've heard this from many
> people
> > who read them - is that they're written at too high a level to be really
> > accessible to lay people.  I thought the point that the study made about
> > the benefit of linking to an "English" dictionary definition of complex
> > terms rather than to another highly technical Wikipedia article was a
> very
> > good one, for example.  We could learn from these studies.
> >
> > Indeed, many science articles are mainly written by professionals in the
> > field (I noted math and physics earlier, but chemistry and of course a
> > large number of computer 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Risker]

In answer to the question of the WMF funding research:
https://meta.wikimedia.org/wiki/Research:FAQ

Risker/Anne


On 8 May 2014 01:13, Anthony Cole  wrote:

> Wow.
>
> Wil - you're going to love WikiData.
>
> Phoebe: I  have seen that list of peer-reviewed articles related to
> Wikipedia medical content. I've extracted those related to quality and
> added more from a couple of database searches I did in January and the list
> of 42 (some are letters and there's a conference abstract, though) are
> collapsed on the WikiProject Medicine talk page now under the heading,
> "This thread is notable."
>
> I've read most but not all of those and, as Andreas mentioned, most of
> those suffered from small sample size and poor or opaque sample selection
> criteria.
>
> Erik, thank you for pointing to the "reviewer" trial. I had read it before
> and I'm glad to have this opportunity to tell you how much I love it. There
> is a big hole in Wikipedia where expert reviewing belongs.
>
> I'm presently on the board of WikiProject Med Foundation, but will be
> stepping down after Wikimania. I mostly edit medical content. Anne is
> right, it is heavily curated. But stuff slips through the net of patrollers
> from time to time, and barely a day goes by without some howler of a
> long-term problem coming to light.
>
> I would like to know - know, rather than rely on my gut feeling - how
> accurate our medical content is. To know that, I think the first step would
> be to get an expert on scientific study design to review the 30-40 existing
> studies that address the quality of our medical content, and tell us what,
> if anything, we can take from that prior work - essentially what Anne
> recommends above, but rather than making my own incompetent and heavily
> biased assessment, get an expert to do it.
>
> My own, inexpert, belief is that those studies are (mostly) so hopelessly
> flawed that nothing can seriously be generalised from them. If I'm right,
> I'd then like us all to consider seriously doing a survey whose design is
> sufficiently rigorous to give us an answer.
>
> Thanks for your thoughts and attention everyone.
>
> Anthony Cole 
>
>
>
> On Thu, May 8, 2014 at 11:05 AM, Andreas Kolbe  wrote:
>
> > On Thu, May 8, 2014 at 3:41 AM, Risker  wrote:
> >
> > > Yes, of course readability analysis is done by automation.  I've yet to
> > > find a consistent readability assessment that doesn't use automation.
> >  It's
> > > not an area where subjectivity is particularly useful.
> > >
> > > And that was an average of 18 minutes per article, i.e., 36 minutes: 18
> > > minutes for the WP article and 18 minutes for the PDQ article.  How
> long
> > do
> > > you really think it should take?  I read several of the articles in
> > under 5
> > > minutes on each site.  Of course, the reviewers wouldn't need to look
> up
> > > the definitions of a lot of the terms that lay people would need to
> look
> > > at, because they were already professionally educated in the topic
> area,
> > so
> > > that would significantly reduce the amount of time required to assess
> the
> > > article.
> > >
> >
> >
> > It took me more than 18 minutes to write the last e-mail in this thread.
> :)
> >
> > The lung cancer article, for example, which was among those reviewed, has
> > well over 4,500 words of prose, and cites 141 references. That's a
> > reviewing speed of 250 words per minute. I don't know if you have ever
> done
> > an FA review ...
> >
> >
> >
> > > Andreas, you seem to have pre-determined that Wikipedia's medical
> > articles
> > > are all terrible and riddled with errors.
> >
> >
> >
> > And I think you are being needlessly defensive. I have an open mind as to
> > what the results might be. What I am sure of is that neither you nor I
> nor
> > the Foundation really know how reliable they are. Why not make an effort
> to
> > find out?
> >
> >
> >
> > > Realistically, they're amongst
> > > the most likely to receive professional editing and review -
> Wikiproject
> > > Medicine does a much better job than people are willing to credit them.
> > >
> >
> >
> > Yes, and many editors there are sorely concerned about the quality of
> > medical information Wikipedia provides to the public.
> >
> > Incidentally, there was a discussion of the JAOA study in The Atlantic
> > today:
> >
> >
> >
> http://www.theatlantic.com/health/archive/2014/05/can-wikipedia-ever-be-a-definitive-medical-text/361822/
> >
> > A member of WikiProject Medicine is quoted in it, as is the study's
> author.
> >
> > —o0o—
> >
> > So both sides acknowledge: There are errors in Wikipedia’s health
> articles.
> > And that’s a problem, because people use them.
> >
> > —o0o—
> >
> >
> >
> >
> >
> > > The biggest weakness to the articles - and I've heard this from many
> > people
> > > who read them - is that they're written at too high a level to be
> really
> > > accessible to lay people.  I thought the point that the study m

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Anders Wennersten]

For sv:wp we only look at quality and reliability (and coverage) only 
for specific subjects areas. I am very skeptical of the value of a 
general study as we already know we are awfully weak in many areas, like 
geographic entities in African countries.

Some examples of our findings
*Swedish adm unts: 100% or close to for coverage, reliability and 
quality in articles. Also are we working to establish a close link to 
Wikidata, to make sure all other version can get the same results in 
corresponding articles
*Birds that are present in Sweden: also here 100% for coverage, 
reliability and quality in articles. This because of a dedicated, 
competent and enthusiastic workgroup
*Medicin.  One of our problem areas. Also it is important to know the 
Swedish government has since long uphold a very qualified webbbased 
information base of health related subjects, so we do not want the svwp 
in any way to be in contradiction of this "official" info. We also found 
problems with articles developed in the medicin project as they had 
recommendation of thing like penicilin use that different from what 
Swedish medicine practise said. We solved this partly buy removing part 
of this info, but to generalize on this get absurd, make it shorter then 
quality and reliabilty goes up


So I wonder if you base assumption of studies of "reliability of 
Wikipedia's content" actually are relevant


Anders




Anthony Cole skrev 2014-05-07 22:17:

Could someone please point me to all the studies the WMF have conducted
into the reliability of Wikipedia's content? I'm particularly interested in
the medical content, but would also like to look over the others too.
Cheers.

Anthony Cole 
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 




___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Anthony Cole]

Thanks Anne.

Anthony Cole 



On Thu, May 8, 2014 at 1:44 PM, Risker  wrote:

> In answer to the question of the WMF funding research:
> https://meta.wikimedia.org/wiki/Research:FAQ
>
> Risker/Anne
>
>
> On 8 May 2014 01:13, Anthony Cole  wrote:
>
> > Wow.
> >
> > Wil - you're going to love WikiData.
> >
> > Phoebe: I  have seen that list of peer-reviewed articles related to
> > Wikipedia medical content. I've extracted those related to quality and
> > added more from a couple of database searches I did in January and the
> list
> > of 42 (some are letters and there's a conference abstract, though) are
> > collapsed on the WikiProject Medicine talk page now under the heading,
> > "This thread is notable."
> >
> > I've read most but not all of those and, as Andreas mentioned, most of
> > those suffered from small sample size and poor or opaque sample selection
> > criteria.
> >
> > Erik, thank you for pointing to the "reviewer" trial. I had read it
> before
> > and I'm glad to have this opportunity to tell you how much I love it.
> There
> > is a big hole in Wikipedia where expert reviewing belongs.
> >
> > I'm presently on the board of WikiProject Med Foundation, but will be
> > stepping down after Wikimania. I mostly edit medical content. Anne is
> > right, it is heavily curated. But stuff slips through the net of
> patrollers
> > from time to time, and barely a day goes by without some howler of a
> > long-term problem coming to light.
> >
> > I would like to know - know, rather than rely on my gut feeling - how
> > accurate our medical content is. To know that, I think the first step
> would
> > be to get an expert on scientific study design to review the 30-40
> existing
> > studies that address the quality of our medical content, and tell us
> what,
> > if anything, we can take from that prior work - essentially what Anne
> > recommends above, but rather than making my own incompetent and heavily
> > biased assessment, get an expert to do it.
> >
> > My own, inexpert, belief is that those studies are (mostly) so hopelessly
> > flawed that nothing can seriously be generalised from them. If I'm right,
> > I'd then like us all to consider seriously doing a survey whose design is
> > sufficiently rigorous to give us an answer.
> >
> > Thanks for your thoughts and attention everyone.
> >
> > Anthony Cole 
> >
> >
> >
> > On Thu, May 8, 2014 at 11:05 AM, Andreas Kolbe 
> wrote:
> >
> > > On Thu, May 8, 2014 at 3:41 AM, Risker  wrote:
> > >
> > > > Yes, of course readability analysis is done by automation.  I've yet
> to
> > > > find a consistent readability assessment that doesn't use automation.
> > >  It's
> > > > not an area where subjectivity is particularly useful.
> > > >
> > > > And that was an average of 18 minutes per article, i.e., 36 minutes:
> 18
> > > > minutes for the WP article and 18 minutes for the PDQ article.  How
> > long
> > > do
> > > > you really think it should take?  I read several of the articles in
> > > under 5
> > > > minutes on each site.  Of course, the reviewers wouldn't need to look
> > up
> > > > the definitions of a lot of the terms that lay people would need to
> > look
> > > > at, because they were already professionally educated in the topic
> > area,
> > > so
> > > > that would significantly reduce the amount of time required to assess
> > the
> > > > article.
> > > >
> > >
> > >
> > > It took me more than 18 minutes to write the last e-mail in this
> thread.
> > :)
> > >
> > > The lung cancer article, for example, which was among those reviewed,
> has
> > > well over 4,500 words of prose, and cites 141 references. That's a
> > > reviewing speed of 250 words per minute. I don't know if you have ever
> > done
> > > an FA review ...
> > >
> > >
> > >
> > > > Andreas, you seem to have pre-determined that Wikipedia's medical
> > > articles
> > > > are all terrible and riddled with errors.
> > >
> > >
> > >
> > > And I think you are being needlessly defensive. I have an open mind as
> to
> > > what the results might be. What I am sure of is that neither you nor I
> > nor
> > > the Foundation really know how reliable they are. Why not make an
> effort
> > to
> > > find out?
> > >
> > >
> > >
> > > > Realistically, they're amongst
> > > > the most likely to receive professional editing and review -
> > Wikiproject
> > > > Medicine does a much better job than people are willing to credit
> them.
> > > >
> > >
> > >
> > > Yes, and many editors there are sorely concerned about the quality of
> > > medical information Wikipedia provides to the public.
> > >
> > > Incidentally, there was a discussion of the JAOA study in The Atlantic
> > > today:
> > >
> > >
> > >
> >
> http://www.theatlantic.com/health/archive/2014/05/can-wikipedia-ever-be-a-definitive-medical-text/361822/
> > >
> > > A member of WikiProject Medicine is quoted in it, as is the study's
> > author.
> > >
> > >

Re: [Wikimedia-l] Metrics - accuracy of Wikipedia articles

[Gerard Meijssen]

Hoi,
When you consider that many if not most US mayors of a city with over
25.000 inhabitants of the 19th century have a Wikipedia article, it is
relevant to notice that most South African members of the National
Assemblee do not have an article [1].

When you consider that English is a major language in South Africa it says
a lot about the bias of Wikipedia. It also does not take much effort to
bring this information to you. It took several hours of adding them to
Wikidata. It is also far from complete. It does however make the point.

Yes, we can spend money on researching the quality of the English Wikipedia
in a narrow band and, yes health information is important but there is a
concerted effort under way to maintain a high quality of information. The
most it will do is provide more information about things we more or less
already know.

The question that I would like to raise is: how are we going to make
available the information that is dormant in Wikipedia? The aim of the WMF
is after all "sharing the sum of all knowledge"...
Thanks,
 GerardM

[1]
http://tools.wmflabs.org/wikidata-todo/autolist.html?q=CLAIM[39%3A16744266]


On 8 May 2014 08:21, Anders Wennersten  wrote:

> For sv:wp we only look at quality and reliability (and coverage) only for
> specific subjects areas. I am very skeptical of the value of a general
> study as we already know we are awfully weak in many areas, like geographic
> entities in African countries.
> Some examples of our findings
> *Swedish adm unts: 100% or close to for coverage, reliability and quality
> in articles. Also are we working to establish a close link to Wikidata, to
> make sure all other version can get the same results in corresponding
> articles
> *Birds that are present in Sweden: also here 100% for coverage,
> reliability and quality in articles. This because of a dedicated, competent
> and enthusiastic workgroup
> *Medicin.  One of our problem areas. Also it is important to know the
> Swedish government has since long uphold a very qualified webbbased
> information base of health related subjects, so we do not want the svwp in
> any way to be in contradiction of this "official" info. We also found
> problems with articles developed in the medicin project as they had
> recommendation of thing like penicilin use that different from what Swedish
> medicine practise said. We solved this partly buy removing part of this
> info, but to generalize on this get absurd, make it shorter then quality
> and reliabilty goes up
>
> So I wonder if you base assumption of studies of "reliability of
> Wikipedia's content" actually are relevant
>
> Anders
>
>
>
>
> Anthony Cole skrev 2014-05-07 22:17:
>
>  Could someone please point me to all the studies the WMF have conducted
>> into the reliability of Wikipedia's content? I'm particularly interested
>> in
>> the medical content, but would also like to look over the others too.
>> Cheers.
>>
>> Anthony Cole 
>> ___
>> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
>> wiki/Mailing_lists/Guidelines
>> Wikimedia-l@lists.wikimedia.org
>> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
>> 
>>
>
>
> ___
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,