[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Lydia_Pintscher added a comment. Ordering by relevancy is a good use case. But I believe relevancy is about more than page views. We have T143424 to come up with a good measure for relevancy that can be used in queries as well. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

christophbraun added a comment. @Nuria WDQS is currently used by the GLAM community to create queries that are beyond the scope of existing tools for a specific purpose as mentioned above. Page views for Wikidata items as well as page views for media files and articles linked to Wikidata

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Yair_rand added a comment. Most query results sets meant for human consumption would benefit from having the results sorted by pageviews. Needing to filter for a certain level of prominence is very common, and using the API isn't a workable solution for most people who would benefit from

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Zache added a comment. One clear use case for Wikimedia editors who aren't coder but who can write/modify SPARQL queries is to sort and filter Petscan and Listeria results. TASK DETAIL https://phabricator.wikimedia.org/T174981 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Gehel added a comment. This is not about disk size, or number of bytes, it is about adding complexity to a system that already isn't stable. As @Nuria was saying, if we go back to a use case, we might find a way to provide a solution. I'm pretty sure that WDQS isn't the solution here. TASK

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Nuria added a comment. I think before talking about bytes you need a use case, what is the use case here? As we mentioned earlier the GLAM folks care about human pageviews (real eye balls) on media files and pages, both cases are (and will be better) satisfied by existing analytics APIs.

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Yurik added a comment. @Gehel lets define `this amount of data`, just for clarity. My back-of-the-envelope calculations: - each pageview statistics statement is a counter (8 bytes), a reference to the name of the article (8 bytes), and property (8 bytes). In reality it might be a bit

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Gehel added a comment. Adding this amount of data to WDQS does not seem to be a good idea. We might want to redefine the higher level problem that we are trying to address here, and maybe implement it in a different way. TASK DETAIL https://phabricator.wikimedia.org/T174981 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Nuria added a comment. Please see: https://stats.wikimedia.org/v2/#/wikidata.org/reading/total-page-views/normal|bar|2-year|agent~user*spider|monthly TASK DETAIL https://phabricator.wikimedia.org/T174981 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Nuria added a comment. @christophbraun I think it would help to start a ticket describing your use case in detail. Have in mind that pageviews (defined as content consumed by humans) do not really "apply" to wikidata items. The bulk of the activity on the site around http requests has a lot

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

christophbraun added a comment. Thanks for your comment @Yurik and @Nuria. The GLAM use case applies to queries beyond the scope of existing tools like https://tools.wmflabs.org/glamtools/treeviews/ or https://tools.wmflabs.org/glamtools/glamorgan.html WDQS would allow us to select a

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Nuria added a comment. Updating WDQS (a relational query engine) with metadata about pageviews (per definition a timeseries) seems not the best idea from a data modeling standpoint. The GLAM use case is much better served by an API that returns pageviews across time, I would put the

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Yurik added a comment. I would guess this is mostly a devops task - orchestrate execution of an updating script. Here's the working implementation - https://github.com/Sophox/sophox/blob/master/osm2rdf/updatePageViewStats.py Simply run it locally near the Blazegraph server. TASK

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

christophbraun added a comment. Thanks for your input @Elya, @Yurik and @Tagishsimon. Do you know who has to greenlight/authorise the upload to the Blazegraph index? Assuming there is a huge backlog for this kind of requests, where can I find it and how is it prioritised? TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Yurik added a comment. @Tagishsimon this proposal would not edit wikidata. Instead, as part of the WDQS import process, it would upload pageviews in bulk from the pageview dump files directly into the Blazegraph index. It could do it every hour, and computation-wise it will be relatively

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Tagishsimon added a comment. Wikidata currently gets ~660k edits per day. This proposal - if I understand it properly - requires an additional ~5 Million edits per day, or, perhaps 5 Millon edits per hour ("and increment the counters once an hour") ... who knows. And that gives us

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Elya added a comment. I'm still not sure what possibilities can be achieved here, but it looks like a whole lot of new things to explore and analyse. One thing of course is the GLAM cooperations @christophbraun mentioned, but I'm sure there will be a lot of ideas … edit-a-thons, etc.

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

christophbraun added a comment. Adding page views to WDQS would be highly beneficial to the GLAMwiki community. Exploring the impact of cultural partnerships with galleries, libraries, archives and museums with a tool beyond the capability of

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Bovlb added a comment. @Yurik asked: I would like to solicit more community feedback on how useful this would be. I would find this extremely useful. What can I do to help make this happen?TASK DETAILhttps://phabricator.wikimedia.org/T174981EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Esc3300 added a comment. There is https://tools.wmflabs.org/glamtools/treeviews/TASK DETAILhttps://phabricator.wikimedia.org/T174981EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Esc3300Cc: EBernhardson, Esc3300, Lydia_Pintscher, Yair_rand, Smalyshev,

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Yurik added a comment. I would like to solicit more community feedback on how useful this would be. Perhaps this is not needed at all, or not worth the hassle As an already working example on a test server, here is a query that lists Wikidata items without French labels but with French

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Smalyshev added a comment. Personally I am not convinced this is a good match for a graph database. This looks like something that is better as a generic API/database. But I am not sure I understand the use-case properly as of yet. At this point, the only way to rank various Wikidata results is

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Yurik added a comment. @Lydia_Pintscher, having a built in ranking system is awesome, but that's a problem of search optimization - just like the other ticket suggests, it will be a part of the search drop-down. Exposing raw views value via wdqs is very different - it allows query authors to

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

Esc3300 added a comment. The simplicity of this approach seems convincing. It could complement number of sitelinks and statements.TASK DETAILhttps://phabricator.wikimedia.org/T174981EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Esc3300Cc: Esc3300,