On Wed, Jan 27, 2016 at 5:15 PM, Marcel Ruiz Forns <[email protected]> wrote: > Hi analytics list, > > In the past months the WikimediaBot convention has been mentioned in a > couple threads, but we (Analytics team) never finished establishing and > advertising it. In this email we explain what the convention is today and > what purpose it serves. And also ask for feedback to be sure we can continue > with the next steps. > > What is the WikimediaBot convention? > It is a way of better identifying Wikimedia traffic originated by bots. > Today we know that a significant share of Wikimedia traffic comes from bots. > We can recognize a part of that traffic with regular expressions[1], but we > can not recognize all of it, because some bots do not identify themselves as > such. If we could identify a greater part of the bot traffic, we could also > better isolate the human traffic and permit more accurate analyses. > > Who should follow the convention? > Computer programs that access Wikimedia sites or the Wikimedia API for > reading purposes* in a periodic, scheduled or automatically triggered way. > > Who should NOT follow the convention? > Computer programs that follow the on-site ad-hoc commands of a human, like > browsers. And well known spiders that are otherwise recognizable by their > well known user-agent strings. > > How to follow the convention? > The client's user-agent string should contain the word "WikimediaBot". The > word can be anywhere within the user-agent string and is case-sensitive. > > So, please, feel free to post your comments/feedback on this thread. In the > course of this discussion we can adjust the convention's definition and, if > no major concerns are raised, in 2 weeks we'll create a documentation page > in Wikitech, send an email to the proper mailing lists and maybe write a > blog post about it. > > Thanks a lot! > > (*) There is already another convention[2] for bots that EDIT Wikimedia > content. > > [1] > https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/Webrequest.java#L49 > [2] https://www.mediawiki.org/wiki/Manual:Bots
The number one User-Agent for hits to api.php is "-" despite [[:meta:User-Agent_policy]] [3]. I think we can do a bit of outreach and get some bot maintainers and maybe a framework or two to implement this. I would be very surprised if it went much further than that unless we actually implement some sort of enforcement policy. That being said I completely understand that keeping a regex or other matching system isn't scalable at all and this convention seems like a reasonable way to start chipping away at the problem. [3]: https://meta.wikimedia.org/wiki/User-Agent_policy -- Bryan Davis Wikimedia Foundation <[email protected]> [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855 _______________________________________________ Analytics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/analytics
