[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

Manuel closed this task as "Resolved". Manuel added a comment. Thank you! TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE, Manuel Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE added a comment. Updated the above comment with a second run and also ran a query for the total IPs for the given period, with the result being `2,115,166`. Percent Scholia queries for the period is thus `28918 / 2115166 * 100`, or 1.37%. TASK DETAIL https://phabricator.w

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE added a comment. A follow up request from @Manuel on this was for the total IPs that are accessing Scholia. The following query was run for this: SELECT count( DISTINCT CASE WHEN query LIKE '%# tool: scholia%' THEN http.client_ip

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE moved this task from Prioritized backlog to Product verification on the Wikidata Analytics (Kanban) board. AndrewTavis_WMDE added a comment. Credit on checking the queries goes to @dcausse :) Added the percent that are identified via a user agent to the results summary just n

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

Manuel added a comment. Hi Andrew, good idea to investigate the types of queries per source! The results seem highly relevant: Could you please add the % of user agent based queries to the results summary? TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://p

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE added a comment. Having derived quick samples (`DISTRIBUTE BY rand()` to mix it up, but nothing more), what I'm seeing is that the comment queries look to be very similar to one another regardless of if they're spiders or non-spiders. Could be that what we're thinking of as

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE added a comment. Quick counts as in the sampling task to check uniqueness of queries and HTTP statuses (I don't think that other measures like variance over weeks, duration or char size would add much). Note that percentages below are for the sub-groups, not for all Scholia

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE added a comment. Results from the following query to check automate traffic via isSpiderUDF is that `91.36%` of the `#tool:

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

dcausse added a comment. In T353453#9524925 , @AndrewTavis_WMDE wrote: > Quick note on this: > > There are two ways that need to be factored in to deriving if a query is from Scholia. Some queries do start with `#tool: scholia` as @d

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE changed the task status from "Open" to "In Progress". AndrewTavis_WMDE triaged this task as "Medium" priority. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc:

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE added a comment. Here are some initial results for consideration. Using the following query over the full dataset from `event.wdqs_external_sparql_query` (last 90 days): SELECT count(*) AS total_scholia_queries FROM event.wdqs_external_sparql_qu

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE added a comment. Quick note on this: There are two ways that need to be factored in to deriving if a query is from Scholia. Some queries do start with `#tool: scholia` as @dcausse suggested, but I checked for user agents and also found that the string `"Scholia"` is also

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE added a comment. Task is refined and I'm starting work on it now. I'm assuming that `event.wdqs_external_sparql_query` is what I'd use for this, and thus we'd be getting aggregate/percent values within a 90 day period given the retention policy :) Let me know if there's

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

AndrewTavis_WMDE claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1, karapayne

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

Manuel edited projects, added Wikidata Analytics (Kanban); removed Wikidata Analytics. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Manuel Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

Manuel added a subscriber: Lydia_Pintscher. Manuel added a comment. Thank you, David! TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Manuel Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Dann

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

Manuel edited parent tasks, added: T337799: [EPIC] Analytics support around splitting the WDQS graph [up to milestone 3]; removed: T349512: [Analytics] Collect multiple sets of SPARQL queries. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikime

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

Manuel renamed this task from "[Analytics] QUERY-Q3: Extract a set of queries known to be used by scholia" to "[Analytics] Impact of Scholia on WDQS". Manuel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.or

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

Manuel added a parent task: T337799: [EPIC] Analytics support around splitting the WDQS graph [up to milestone 3]. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Manuel Cc: Aklapper, Manuel, Dann

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

Manuel created this task. Manuel added projects: Wikidata Analytics, Wikidata. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Scope - How many SPARQL queries are related to Scholia Notes - - Scholia queries shoul

[Wikidata-bugs] [Maniphest] T353453: [Analytics] Impact of Scholia on WDQS

Manuel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T353453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Manuel Cc: Aklapper, Manuel, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Itama