Hey there,
I sympathize with your dilemma...and I think we might have actually
talked about this at Wikimania 2009. Unfortunately, while you may be
satisfied that 600 journal articles + theses is enough (I certainly
would be too), you should be equipped to recognize that if you keep it
that way you are systematically excluding large, significant bodies of
research deriving from computer science and HCI. As you make this
choice, read through one or two of these conference papers and measure
it against the quality of a randomly selected set of journal articles in
your set:
- http://dub.washington.edu/djangosite/media/papers/tmpZ77p1r.pdf
-
http://www.aaai.org/ocs/index.php/ICWSM/ICWSM10/paper/download/1485/1841
- http://www.cs.cornell.edu/~danco/research/papers/suggestbot-iui07.pdf
- http://users.soe.ucsc.edu/~luca/papers/07/wikiwww2007.pdf
- http://portal.acm.org/citation.cfm?id=1518928
I bet that these conference papers are on the balance of higher quality
than a random journal article in your set.
Unfortunately, there isn't a good answer for the best methods to follow.
Everyone in my field (HCI) pretty much knows what the first tier
conferences are where wikipedia research is published: CHI, CSCW, and
UIST; and second tier at GROUP. These are all under the ACM SIGCHI
banner (http://www.sigchi.org/). Another way to put this is that there
are no objective measures, its a question of what the researchers
themselves see as high quality. Ultimately, this is the same as with
journals, although they tend to have impact factors. If I were to
estimate how many high quality conference papers from the HCI angle
there are, I would put it at about 20-30.
Of course, this is only for HCI research, not all CS research.
Conferences such as WWW have published excellent research on Wikipedia,
such as the initial paper out of the WikiTrust group, which, if you've
been around the wiki community, know that they have had a big impact.
WWW is considered to be a high quality CS conference. Likewise, there
has been Wiki research published at database and AI conferences. For
example, the Intelligence in Wikipedia project (summarized here
http://portal.acm.org/citation.cfm?id=1620344).
Unfortunately, your two links to top conferences are pretty much
inaccurate pictures of the CS conference field (for example, the deakin
link puts GECCO as the top conference in one of the major categories,
which is basically laughable). And while we might all love wikisym, it
from an academic standpoint, it is definitely not a tier one venue.
I cringe to suggest this, but one possible methodology you might follow
is to do citation count filtering, using, e.g. google scholar. Citations
give you an indicator of whether other researchers have found it useful
to draw on. Look at the average citation count of the journal papers,
then filter your list of 1500 conference papers down to those papers
that have, say, twice the citations as the average citation count of a
journal article.
Honestly though, your best methodology would be to have a small group of
HCI researchers, a small group of AI researchers, and a small group of
database researchers who have worked on wikipedia compile a list of the
conference papers that they believe are best representative of the
research that that community has done on wikipedia.
Hope that helps, and sorry to hear you still struggling with this issue.
Best,
Travis
On 3/15/11 11:56 AM, Chitu Okoli wrote:
> James and Travis, you bring up a point that we have struggled back and
> forth with for several months. We really, really would like to include
> conference articles, but we just can't see how we could handle many more
> articles than what we've got now. We've been working on and off on this
> project for over two years now. (You can find works in progress at the
> link at the bottom to my website.) We'd like to get it done eventually,
> and we can only handle so many articles.
>
> We considered including top-tier conferences, but the question is, what
> is a "top conference"? In trying to answer this, we looked at a couple
> of sources:
> * Top Tier and 2nd tier conferences from
> http://webdocs.cs.ualberta.ca/~zaiane/htmldocs/ConfRanking.html
> * A-ranked conferences in Information and Computing Sciences from
> http://lamp.infosys.deakin.edu.au/era/?page=cforsel10
> * We also considered including all WikiSym articles on Wikipedia
>
> We identified which of the 1,500 conference papers from
> http://en.wikipedia.org/wiki/User:Moudy83/conference_papers were "top
> conferences" by those definitions, and we found over 400. On top of our
> 600 journal articles and doctoral theses, we think 1,000 papers is just
> too much for us to handle.
>
> If we could somehow narrow it down to 100 relevant conference papers, we
> could add that in, but no more. However, how do we select which
> conferences are "must includes" while unfortunately leaving out the
> rest? We just don't know how to do this in a non-arbitrary, objective
> manner that would truly identify the top 100 conference papers on
> Wikipedia that contribute to scholarly knowledge.
>
> Any ideas on how to do this would be very much appreciated.
>
> Regards,
> Chitu
>
>
>
> -------- Message original --------
> Sujet: Re: [Wiki-research-l] Request to verify articles for Wikipedia
> literature review
> De : Travis Kriplean <tra...@cs.washington.edu>
> Pour : Research into Wikimedia content and communities
> <wiki-research-l@lists.wikimedia.org>
> Date : 14/03/2011 3:46 PM
>> As an HCI/CS researcher who has published at top peer-reviewed
>> conferences about Wikipedia, but not journals, I'd like to echo James'
>> statements. Journals are not the norm in CS/HCI research. Knowledge is
>> shared through conferences, not journals.
>>
>> On 3/14/11 11:32 AM, James Howison wrote:
>>> Hi there,
>>>
>>> Great project; massive but will be much appreciated. We did something
>>> similar for empirical studies of Open Source, recently accepted at
>>> ACM Computing Surveys (PDF pre-print available here [1], article not
>>> in print until 2012 (!! that's another email entirely, bah))
>>>
>>> I recognize the need to cut down the number of articles for review,
>>> we reviewed around 600 and that was a multi-year effort. We did that
>>> mainly by excluding conceptual (hence empirical) or passing reference
>>> articles (ie we did a two-step filter on many more articles), but
>>> were forced to only do journal articles for updates during the (long)
>>> revision process. I regret that necessity, it decreases the utility
>>> of the work.
>>>
>>> Given the publication venues of choice for many academics in this
>>> community I do wonder if you aren't shooting yourself in the foot by
>>> excluding peer-reviewed conferences and restricting to journals.
>>> Personally I'd rather read a review that included the top journals
>>> and top conferences than one that included all journals. Or even
>>> rather read a review over a shorter time period that included
>>> publications over journals and conferences, or on more specified
>>> topics. The interesting question is, "what do we know about
>>> wikipedia" not "what did we publish in journals about wikipedia". In
>>> particular you will find you have systematically excluded the
>>> contribution of HCI authors.
>>>
>>> Given the commendable and massive effort you are providing (and your
>>> approach to coverage below is really interesting), getting that wrong
>>> at the outset seems a shame.
>>>
>>> Best regards,
>>> James Howison
>>>
>>> [1] Crowston, K., Wei, K., Howison, J., and Wiggins, A. (2012). Free
>>> (libre) open source software development: What we know and what we do
>>> not know. ACM Computing Surveys, 44(2):
>>> http://floss.syr.edu/content/freelibre-open-source-software-development-what-we-know-and-what-we-do-not-know
>>>
>>>
>>>
>>> On Mar 14, 2011, at 13:58, Chitu Okoli wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> We are a research group conducting a systematic literature review on
>>>> Wikipedia-related peer-reviewed academic studies published in the
>>>> English language. (Although there are many excellent studies in
>>>> other languages, we unfortunately do not have the resources to
>>>> systematically review these at any kind of acceptable scholarly
>>>> level. Also, our study is about Wikipedia only, not about other
>>>> Wikimedia Foundation projects. However, we do include studies about
>>>> other language Wikipedias, as long as the studies are published in
>>>> English.) We have completed a search using many major databases of
>>>> scholarly research. In a separate thread, we will also talk about
>>>> research questions related to our review.
>>>>
>>>> Thanks for your help.
>>>>
>>>> Chitu Okoli, Concordia University, Montreal, Canada
>>>> (http://chitu.okoli.org/professional/open-content/wikipedia-and-open-content.html)
>>>>
>>>> Arto Lanamäki, University of Agder, Kristiansand, Norway
>>>> Mohamad Mehdi, Concordia University, Montreal, Canada
>>>> Mostafa Mesgari, Concordia University, Montreal, Canada
>
>
>
> _______________________________________________
> Wiki-research-l mailing list
> Wiki-research-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
