Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 14 July 2011 00:40, Howie Fung hf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without sockpuppetry. On average our articles get one or two edits a month, many get far less. I would not be surprised if 100,000 of the 374k in the trial had less than ten ratings even if trialled for a couple of months. Lastly we need to be prepared for sockpuppetry, especially as these are random unsigned votes with no rationale. Can we have assurances that something is being built into the scheme to combat this? Regards WereSpielChequers On 14 July 2011 10:08, David Gerard dger...@gmail.com wrote: On 14 July 2011 00:40, Howie Fung hf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
A couple of fair points. However, I would disagree that everyone is interested in editing or improving the encyclopedia; some are perfectly content on reading the content therein and, if given the chance, say what they think about out (not necessarily on Wikipedia, but could be anywhere on the Web). I mean, we cannot point a gun to their head and make them edit something, as this is a purely volunteer project. However, you've made a good point there about gaming the system and intentionally trying to garner high ratings. For example, one could create a horrid piece of crap article which would have no chance of staying on Wikipedia and canvass his/her buddies to flood said piece of crap with 5.0's across the board. This thing precisely happens from time to time on YouTube. I don't know how this could be prevented, but I acknowledge that even this feedback system, as with all others, are not perfect and comes with systemic flaws. -MuZemike On 7/14/2011 7:56 AM, WereSpielChequers wrote: Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without sockpuppetry. On average our articles get one or two edits a month, many get far less. I would not be surprised if 100,000 of the 374k in the trial had less than ten ratings even if trialled for a couple of months. Lastly we need to be prepared for sockpuppetry, especially as these are random unsigned votes with no rationale. Can we have assurances that something is being built into the scheme to combat this? Regards WereSpielChequers On 14 July 2011 10:08, David Gerarddger...@gmail.com wrote: On 14 July 2011 00:40, Howie Funghf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 14 July 2011 18:01, MuZemike muzem...@gmail.com wrote: However, you've made a good point there about gaming the system and intentionally trying to garner high ratings. For example, one could create a horrid piece of crap article which would have no chance of staying on Wikipedia and canvass his/her buddies to flood said piece of crap with 5.0's across the board. This thing precisely happens from time to time on YouTube. I don't know how this could be prevented, but I acknowledge that even this feedback system, as with all others, are not perfect and comes with systemic flaws. There are various ways to mitigate these effects, e.g. cut off the top and bottom 10% of ratings when calculating the displayed numbers. But the essential problem is [[Goodhart's law]]: once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting policy, then it will lose the information content that would qualify it to play such a role. So the answer is not to take the ratings *too* seriously for purposes of writing the encyclopedia. - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
Cutting off the top or bottom 10% wouldn't work if 4chan targets the articles written by one of our editors, if anything the non4chan votes will be in the top 10% that you discard. To be honest I'm not particularly worried if people canvass their mates to give straight 5s to an obscure article that only a few hundred people will ever notice. I would anticipate that will happen whenever someone files an AFD on an article that is of interest to a particular fansite, and if anything it will be less disruptive to have a bunch of fans boost the articles ratings than it will be to deal with those same fans at the AFD. The positive ratings that really matter to editors on this site are things like FA and GA and I don't see this system replacing that. I'm more concerned that this will give people an underhand way to get back at an editor they dislike. Unless I'm missing something and this has already been anticipated, this system needs a mechanism to spot when a group of editors anonymously rate everything another editor has done as rubbish. WSC On 14 July 2011 18:01, MuZemike muzem...@gmail.com wrote: A couple of fair points. However, I would disagree that everyone is interested in editing or improving the encyclopedia; some are perfectly content on reading the content therein and, if given the chance, say what they think about out (not necessarily on Wikipedia, but could be anywhere on the Web). I mean, we cannot point a gun to their head and make them edit something, as this is a purely volunteer project. However, you've made a good point there about gaming the system and intentionally trying to garner high ratings. For example, one could create a horrid piece of crap article which would have no chance of staying on Wikipedia and canvass his/her buddies to flood said piece of crap with 5.0's across the board. This thing precisely happens from time to time on YouTube. I don't know how this could be prevented, but I acknowledge that even this feedback system, as with all others, are not perfect and comes with systemic flaws. -MuZemike On 7/14/2011 7:56 AM, WereSpielChequers wrote: Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without sockpuppetry. On average our articles get one or two edits a month, many get far less. I would not be surprised if 100,000 of the 374k in the trial had less than ten ratings even if trialled for a couple of months. Lastly we need to be prepared for sockpuppetry, especially as these are random unsigned votes with no rationale. Can we have assurances that something is being built into the scheme to combat this? Regards WereSpielChequers On 14 July 2011 10:08, David Gerarddger...@gmail.com wrote: On 14 July 2011 00:40, Howie Funghf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
WereSpielChequers, thanks for the great feedback. We are going to analyze the overall effect of AFT on article edit volume. More generally, for all retention features we are currently deploying, we will be studying both how they affect edit activity at article-level and how they affect individual editor contributions. Updates will be posted as usual on the AFT research page http://www.mediawiki.org/wiki/Article_feedback/Research Dario On Jul 14, 2011, at 5:56 AM, WereSpielChequers wrote: Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without sockpuppetry. On average our articles get one or two edits a month, many get far less. I would not be surprised if 100,000 of the 374k in the trial had less than ten ratings even if trialled for a couple of months. Lastly we need to be prepared for sockpuppetry, especially as these are random unsigned votes with no rationale. Can we have assurances that something is being built into the scheme to combat this? Regards WereSpielChequers On 14 July 2011 10:08, David Gerard dger...@gmail.com wrote: On 14 July 2011 00:40, Howie Fung hf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On Jul 14, 2011, at 10:11 AM, David Gerard wrote: On 14 July 2011 18:01, MuZemike muzem...@gmail.com wrote: However, you've made a good point there about gaming the system and intentionally trying to garner high ratings. For example, one could create a horrid piece of crap article which would have no chance of staying on Wikipedia and canvass his/her buddies to flood said piece of crap with 5.0's across the board. This thing precisely happens from time to time on YouTube. I don't know how this could be prevented, but I acknowledge that even this feedback system, as with all others, are not perfect and comes with systemic flaws. There are various ways to mitigate these effects, e.g. cut off the top and bottom 10% of ratings when calculating the displayed numbers. But the essential problem is [[Goodhart's law]]: once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting policy, then it will lose the information content that would qualify it to play such a role. So the answer is not to take the ratings *too* seriously for purposes of writing the encyclopedia. - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 14 July 2011 18:22, WereSpielChequers werespielchequ...@gmail.com wrote: Cutting off the top or bottom 10% wouldn't work if 4chan targets the articles written by one of our editors, if anything the non4chan votes will be in the top 10% that you discard. [...] Unless I'm missing something and this has already been anticipated, this system needs a mechanism to spot when a group of editors anonymously rate everything another editor has done as rubbish. Again, this is a problem of taking the numbers too seriously. There is *no* system *anywhere* that can't be gamed. Before saying we need to deal with exploit x, we should see what exploits actually happen. Making all the rating data publicly available for analysis (with no usernames or IPs attached, of course) is a first step. Before proposing solutions to problems in the data, look at the data ;-) - d. - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On Thu, Jul 14, 2011 at 2:31 PM, David Gerard dger...@gmail.com wrote: Making all the rating data publicly available for analysis (with no usernames or IPs attached, of course) is a first step. Before proposing solutions to problems in the data, look at the data ;-) A sound recommendation from the psychology literature on problem solving. To quote Eliezer Yudkowsky ( http://lesswrong.com/lw/ka/hold_off_on_proposing_solutions/ ) quoting Robyn Dawes (_Rational Choice in an Uncertain World_) expanding Norman R. F. Maier: ...when a group faces a problem, the natural tendency of its members is to propose possible solutions as they begin to discuss the problem. Consequently, the group interaction focuses on the merits and problems of the proposed solutions, people become emotionally attached to the ones they have suggested, and superior solutions are not suggested. Maier enacted an edict to enhance group problem solving: Do not propose solutions until the problem has been discussed as thoroughly as possible without suggesting any. It is easy to show that this edict works in contexts where there are objectively defined good solutions to problems. -- gwern http://www.gwern.net ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On Thu, Jul 14, 2011 at 18:22, WereSpielChequers werespielchequ...@gmail.com wrote: To be honest I'm not particularly worried if people canvass their mates to give straight 5s to an obscure article that only a few hundred people will ever notice. I would anticipate that will happen whenever someone files an AFD on an article that is of interest to a particular fansite, and if anything it will be less disruptive to have a bunch of fans boost the articles ratings than it will be to deal with those same fans at the AFD. The positive ratings that really matter to editors on this site are things like FA and GA and I don't see this system replacing that. I think the important think about the article feedback tool is that hopefully it will allow WikiProjects to prioritise article improvements. Let's say you are involved with WikiProject Philosophy: it'd be really useful to get a list of all the philosophy articles with article feedback statistics mixed in. If we have an article that is getting very variable ratings, going up and down all over the place, that's a useful measure for having passionate readers. If there's an article with organically occurring high ratings from the readers, that is something the WikiProject should collectively consider pushing towards Good Article or Featured Article. The problem is we get the 'Bieber problem': people voting on the basis of their views of the article's subject rather than the article, so people who love Justin Bieber upvote it and people who loathe him downvote it, even though we are asking whether they think the article is good. The negative side is worse here: people downvoting the article as a kind of 'delete' vote - they think that saying the article is poor quality because we are giving too much coverage to a subject we shouldn't be giving coverage to. There is a good side though: we can use the different categories quite usefully. If we have an article that is highly rated in three of the four criteria but not so well rated in another, that's potentially something we could flag up to WikiProjects as an area for improvement. The article feedback tool is just that... a tool we can use to feed back into the project. It shouldn't ever be an end in itself. -- Tom Morris http://tommorris.org/ ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
