[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Umherirrender added a comment. In T296383#7754968 , @Umherirrender wrote: > Its done, the remaining page ids are orphaned page ids (at least on replica) and does not exists in the `page` table > > MariaDB [warwiki_p]> SELECT

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Umherirrender added a comment. In T296383#7764192 , @Lucas_Werkmeister_WMDE wrote: > We somehow ended up with less total entity usage rows than we started with, I’m not sure why that would be the case. 路 On the grafana dashboard it

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE moved this task from Peer Review to Our work done on the Wikidata-Campsite (Team A Hearth ) board. Lucas_Werkmeister_WMDE added a comment. Confirmed: MariaDB [warwiki]> SELECT NOW() AS asof, COUNT(*) AS total, SUM(eu_aspect LIKE 'C%') AS statements,

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Umherirrender closed this task as "Resolved". Umherirrender added a comment. Its done, the remaining page ids are orphaned page ids (at least on replica) and does not exists in the `page` table MariaDB [warwiki_p]> SELECT eu_page_id, eu_row_id FROM wbc_entity_usage WHERE eu_aspect

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Umherirrender added a comment. MariaDB [warwiki_p]> SELECT NOW() AS asof, COUNT(*) AS total, SUM(page_links_updated < '20220301182633') AS backlog FROM page INNER JOIN templatelinks ON page_id = tl_from WHERE tl_namespace = 10 AND tl_title = 'Wikidata_image';

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Umherirrender added a comment. It is a bit slower (only 267k in 24h, not 352k in 24h): MariaDB [warwiki_p]> SELECT NOW() AS asof, COUNT(*) AS total, SUM(page_links_updated < '20220301182633') AS backlog FROM page INNER JOIN templatelinks ON page_id = tl_from WHERE tl_namespace = 10

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Umherirrender added a comment. Technical done is the jobqueue when the `page_links_updated` timestamp is higher than the start of the recursive link update for all the pages embedding the template. Pages getting a update from another job queue run or the script purge are deduplicated.

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE added a comment. I guess that corresponds to the spike in refreshLinks jobs at 18:36 UTC, though I don’t know how to check when the

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Umherirrender added a comment. The most usage is coming from one template https://war.wikipedia.org/wiki/Batakan:Wikidata_image select count(*) from templatelinks where tl_namespace = 10 and tl_title = 'Wikidata_image' and tl_from_namespace = 0;

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Ladsgroup added a comment. At least on my side, a script running for a month is totally fine, we have maintenance script running for five months. TASK DETAIL https://phabricator.wikimedia.org/T296383 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE moved this task from Tech Verification to Peer Review on the Wikidata-Campsite (Team A Hearth ) board. Lucas_Werkmeister_WMDE added a comment. Moving back to review for someone™ to take a look at that script; if it looks okay, I think we can start running larger

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE added a comment. In T296383#7741822 , @Lucas_Werkmeister_WMDE wrote: > I’ve started putting together a script to do the purge in `~lucaswerkmeister-wmde/purge-other-usages/` on Toolforge. (It’s a Git repository,

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE added a comment. I’ve started putting together a script to do the purge in `~lucaswerkmeister-wmde/purge-other-usages/` on Toolforge. (It’s a Git repository, with some history too.) The script can be run on a Buster bastion using the venv in that directory; it goes

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE added a comment. One month later: MariaDB [warwiki]> SELECT NOW() AS asof, COUNT(*) AS total, SUM(eu_aspect LIKE 'C%') AS statements, SUM(eu_aspect LIKE 'O%') AS others FROM wbc_entity_usage; +-+-++-+ |

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE added a comment. Current status: MariaDB [warwiki]> SELECT NOW() AS asof, COUNT(*) AS total, SUM(eu_aspect LIKE 'C%') AS statements, SUM(eu_aspect LIKE 'O%') AS others FROM wbc_entity_usage; +-+-++-+ | asof

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE added a comment. Quick check on the recentchanges table: so far (i.e. a few days after we deployed the change), it contains some 104k rows, 97k of which come from Wikidata. MariaDB [warwiki]> SELECT COUNT(*) AS total, SUM(rc_type = 5) FROM recentchanges;

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T296383 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: noarave, Maintenance_bot Cc: Ladsgroup, Aklapper, Lydia_Pintscher, hoo, Michael, Marostegui,

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-01-20T12:31:14Z] Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:755322|Enable usage tracking for statements in Waray Wikipedia (T296383 )]] (expecting some

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

gerritbot added a comment. Change 755322 **merged** by jenkins-bot: [operations/mediawiki-config@master] Enable usage tracking for statements in Waray Wikipedia https://gerrit.wikimedia.org/r/755322 TASK DETAIL https://phabricator.wikimedia.org/T296383 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

noarave moved this task from Doing to Peer Review on the Wikidata-Campsite (Team A Hearth ) board. noarave added a comment. After this is deployed we should make sure data is being tracked under "Statement Flags" in the warwiki dashboard:

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T296383 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: noarave, gerritbot Cc: Ladsgroup, Aklapper, Lydia_Pintscher, hoo, Michael, Marostegui, alaa,

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

gerritbot added a comment. Change 755322 had a related patch set uploaded (by Noa wmde; author: Noa wmde): [operations/mediawiki-config@master] Enable usage tracking for statements in Waray Wikipedia https://gerrit.wikimedia.org/r/755322 TASK DETAIL

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

noarave claimed this task. Restricted Application added a project: User-Noarave. TASK DETAIL https://phabricator.wikimedia.org/T296383 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: noarave Cc: Ladsgroup, Aklapper, Lydia_Pintscher, hoo, Michael,

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

noarave added a project: Wikidata-Campsite (Team A Hearth ). TASK DETAIL https://phabricator.wikimedia.org/T296383 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: noarave Cc: Ladsgroup, Aklapper, Lydia_Pintscher, hoo, Michael, Marostegui, alaa,

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Marostegui removed a project: SRE. TASK DETAIL https://phabricator.wikimedia.org/T296383 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Ladsgroup, Aklapper, Lydia_Pintscher, hoo, Michael, Marostegui, alaa, Lucas_Werkmeister_WMDE,

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Marostegui edited projects, added Data-Persistence (Consultation); removed DBA. TASK DETAIL https://phabricator.wikimedia.org/T296383 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Ladsgroup, Aklapper, Lydia_Pintscher, hoo, Michael,

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE added a comment. 2021-11-24: lucaswerkmeister-wmde@stat1007:~$ sudo -u analytics-wmde analytics-mysql warwiki <<< 'SELECT table_name, table_rows FROM information_schema.tables WHERE table_schema = "warwiki" AND table_name IN ("wbc_entity_usage",

[Wikidata-bugs] [Maniphest] T296383: Enable statement usage tracking on warwiki

Lucas_Werkmeister_WMDE created this task. Lucas_Werkmeister_WMDE added projects: MediaWiki-extensions-WikibaseClient, Wikidata, wdwb-tech, DBA, SRE. Restricted Application added a subscriber: alaa. TASK DESCRIPTION As agreed in T188730#7461581