[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Ladsgroup added a comment. Just noting that this is enabled eveyrwhere except commons, hywiki, cebwiki, and warwikiTASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, LadsgroupCc: He7d3r, Marostegui,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-02-28T18:44:05Z] Synchronized wmf-config/InitialiseSettings.php: Reduce the batch size of statment usage tracking to 33 T151717 (duration: 00m 57s)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-02-28T18:42:42Z] Synchronized wmf-config/Wikibase.php: Reduce the batch size of statment usage tracking to 33 T151717 (duration: 00m 57s)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 415264 merged by jenkins-bot: [operations/mediawiki-config@master] Reduce the batch size of statment usage tracking to 33 https://gerrit.wikimedia.org/r/415264TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 415264 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani): [operations/mediawiki-config@master] Reduce the batch size of statment usage tracking to 33 https://gerrit.wikimedia.org/r/415264TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-02-26T14:48:01Z] Synchronized wmf-config/InitialiseSettings.php: [[gerrit:V|Enable statement usage tracking in several wikis (T151717)]] (duration: 00m 57s)TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 414654 merged by jenkins-bot: [operations/mediawiki-config@master] Enable statement usage tracking in several wikis https://gerrit.wikimedia.org/r/414654TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 414654 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani): [operations/mediawiki-config@master] Enable statement usage tracking in several wikis https://gerrit.wikimedia.org/r/414654TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Lydia_Pintscher added a comment. Is there anything left to do here or can it be closed?TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, Lydia_PintscherCc: He7d3r, Marostegui, Bawolff, eranroz, Ottomata,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-10-18T08:41:48Z] Synchronized wmf-config/InitialiseSettings.php: Re-enable Statement usage tracking on cawiki (T151717) (duration: 00m 50s)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 384592 merged by jenkins-bot: [operations/mediawiki-config@master] Re-enable Statement usage tracking on cawiki https://gerrit.wikimedia.org/r/384592TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. I've scheduled the deployment for Wednesday (October 18) 18:00–19:00 UTC. I'll closely monitor the situation after, but I don't expect any more havoc.TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 384592 had a related patch set uploaded (by Hoo man; owner: Hoo man): [operations/mediawiki-config@master] Re-enable Statement usage tracking on cawiki https://gerrit.wikimedia.org/r/384592TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

eranroz added a comment. 1, @hoo so how do we move forward with deployment to larger wikis/wikis with more wikidata usage such as cawiki? Do we have an issue with DB size? (e.g requires more thought on wbc_entity_usage design) or is it disabled in cawiki because it updated too fast? (so we just

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. cawiki had (at the time statement usage were disable again) 90,430 statement usages from just 1,118 pages. This means each of these pages had an average of about 81 statement usages (median: 90) and one page had 196 even. Top 10 pages: mysql:wikiadmin@db1094 [cawiki]> SELECT

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. Table sizes pre-deploy to cawiki and cewiki: mysql:wikiadmin@db1079 [cawiki]> SHOW TABLE STATUS like "wbc_entity_usage"\G *** 1. row *** Name: wbc_entity_usage Engine: InnoDB Version: 10

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-10-11T12:54:16Z] Synchronized wmf-config/InitialiseSettings.php: Disable Statement usage tracking on cawiki (T151717) (duration: 00m 47s)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 383556 merged by jenkins-bot: [operations/mediawiki-config@master] Disable statement usage tracking on cawiki https://gerrit.wikimedia.org/r/383556TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 383556 had a related patch set uploaded (by Hoo man; owner: Hoo man): [operations/mediawiki-config@master] Disable statement usage tracking on cawiki https://gerrit.wikimedia.org/r/383556TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-10-11T12:44:53Z] Synchronized wmf-config/InitialiseSettings.php: (temp) Disable Statement usage tracking on cawiki (T151717) (duration: 00m 48s)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-10-11T12:38:06Z] Synchronized wmf-config/InitialiseSettings.php: Enable Statement usage tracking on cawiki and cewiki (T151717) (duration: 00m 47s)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 383440 merged by jenkins-bot: [operations/mediawiki-config@master] Enable Statement usage tracking on cawiki and cewiki https://gerrit.wikimedia.org/r/383440TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 383439 merged by jenkins-bot: [operations/mediawiki-config@master] Move WB client "disabledUsageAspects" setting into $wmgWikibaseDisabledUsageAspects https://gerrit.wikimedia.org/r/383439TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 383440 had a related patch set uploaded (by Hoo man; owner: Hoo man): [operations/mediawiki-config@master] Enable Statement usage tracking on cawiki and cewiki https://gerrit.wikimedia.org/r/383440TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 383439 had a related patch set uploaded (by Hoo man; owner: Hoo man): [operations/mediawiki-config@master] Move WB client "disabledUsageAspects" setting into $wmgWikibaseDisabledUsageAspects https://gerrit.wikimedia.org/r/383439TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. In T151717#3672577, @jcrespo wrote: 51.5M you meant, maybe? No for kowiki we're indeed talking about "just" a few (~5) million statement usages… after all it has only about 1.5% the number of all+other usages ruwiki has (about 0.5% if we only include all usages).TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. 51.5M you meant, maybe?TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Bawolff, eranroz, Ottomata, PokestarFan, Ladsgroup, Stashbot, gerritbot, Halfak, jcrespo, TomT0m,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. I'll enable the tracking on the two wikis (cawiki, cewiki) tomorrow then. Btw, the estimate for kowiki changed down from 5.5M statement usages to 5.15M already (that's just a increase by a factor of 19.9)… I guess this will go down further as more pages see updates.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. In T151717#3670067, @jcrespo wrote: I have the feeling that these numbers could be meaningless on such small wikis, given the issues on recentchanges with only some large wikis such as commons and ruwiki. Could the same issue happen to those a x10 growth. I know the case is

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. What I mean is that the number are ok to proceeed (not a big deal), but still worried for the large wikis. I know you do not have all the answers, I just was talking aloud.TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. I have the feeling that these numbers could be meaningless on such small wikis, given the issues on recentchanges with only some large wikis such as commons and ruwiki. Could the same issue happen to those a x10 growth. I know the case is different, pages * edits, vs

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. In T151717#3669875, @jcrespo wrote: What you write is ok, but IF you want our opinion, can you translate that into increase of row storage and inserts/other write activity compared to the full table size/previous state? mysql:wikiadmin@db1094 [kowiki]> SHOW TABLE STATUS

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. What you write is ok, but IF you want our opinion, can you translate that into increase of row storage and inserts/other write activity compared to the full table size/previous state?TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. On kowiki currently about 21.7% of all pages having O or X usages have seen a page link update since the deploy. mysql:wikiadmin@db1086 [kowiki]> SELECT COUNT(*) FROM (SELECT COUNT(*) FROM page INNER JOIN wbc_entity_usage ON eu_page_id = page_id WHERE eu_aspect IN ('X', 'O')

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. After about 4 days we have: mysql:wikiadmin@db1076 [trwiki]> SELECT COUNT(*) FROM wbc_entity_usage WHERE eu_aspect LIKE "C.%"; +--+ | COUNT(*) | +--+ | 230846 | +--+ mysql:wikiadmin@db1076 [trwiki]> SELECT COUNT(*) FROM wbc_entity_usage WHERE

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-10-05T13:57:24Z] Synchronized wmf-config/Wikibase-production.php: Enable Statement usage tracking on kowiki and trwiki (T151717) (duration: 00m 50s)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. Table stats on trwiki pre-deploy: No eu_aspect 24 L.en 460 L.tr 130725 O 508625 S 3829T 12367 X mysql:wikiadmin@db1074 [trwiki]> SHOW TABLE STATUS like "wbc_entity_usage"\G *** 1. row ***

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 382414 merged by jenkins-bot: [operations/mediawiki-config@master] Enable Statement usage tracking on kowiki and trwiki https://gerrit.wikimedia.org/r/382414TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 382414 had a related patch set uploaded (by Hoo man; owner: Hoo man): [operations/mediawiki-config@master] Enable Statement usage tracking on kowiki and trwiki https://gerrit.wikimedia.org/r/382414TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. I am a bit lost with the estimation- is that realistic, is the number of usages more or less right with your estimation, or is it going to be a quadratic _expression_ because we do not know accurately the number of statement usages per statement? If the estimations are

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. On elwiki we saw the number of statement usage to be about 5.3 times the number of all + other usages ((1.867e6)/(272.7e3+80.554e3) = 5.28, data taken from Grafana today). For enwiki that would mean about 25 million statement usages (compared to a total of 14.8 million usages

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. Now that all articles have been refreshed (see T151717#3621993/T151717#3621975 for a comparison): +--+ | COUNT(*) | +--+ | 2646007 | +--+ 1 row in set (16.76 sec) ++---+ | No | eu_aspect | ++---+ | 269763 | X

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-09-26T14:26:30Z] mwscript refreshLinks.php --wiki elwiki --namespace 0 on terbium has finished (T151717)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. Populating elwiki will be done in the next few hours, I'll then post updated stats. In order to keep moving fast, I suggest targeting another mid-sized wiki… any suggestions? kowiki and trwiki we're suggested earlier and IMO are good next targets. What do you think

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

eranroz added a comment. In T151717#3624131, @daniel wrote: In T151717#3623996, @Lydia_Pintscher wrote: I assume the latter also means we are not decreasing the number of irrelevant changes shown in the watchlist and recent changes yet? Potentially yes. How much impact this actually has is

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

daniel added a comment. In T151717#3623996, @Lydia_Pintscher wrote: I assume the latter also means we are not decreasing the number of irrelevant changes shown in the watchlist and recent changes yet? Potentially yes. How much impact this actually has is hard to predict. If most pages only use

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Lydia_Pintscher added a comment. I assume the latter also means we are not decreasing the number of irrelevant changes shown in the watchlist and recent changes?TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. (Shortly) after the refresh links got started: mysql:wikiadmin@db1038 [elwiki]> SHOW TABLE STATUS like "wbc_entity_usage"\G *** 1. row *** Name: wbc_entity_usage Engine: InnoDB Version: 10

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. Cool, get if you can some `SHOW TABLE STATUS like stats, to get the "before" state in bytes (even if it had started some time ago).TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. Just started refreshLinks.php for all articles on elwiki (https://wikitech.wikimedia.org/w/index.php?diff=1770578=1770577), this should fully update the entity usage table.TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. Note: Before the deploy, elwiki had 798858 usages only: mysql:wikiadmin@db1038 [elwiki]> SELECT COUNT(*) AS No, eu_aspect FROM wbc_entity_usage GROUP BY eu_aspect ORDER BY No DESC; ++---+ | No | eu_aspect | ++---+ | 270849 | X | |

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. After a few hours (w/o any mass purges from my side), the table looks like this: +--+ | COUNT(*) | +--+ | 1577837 | +--+ 1 row in set (0.50 sec) ++---+ | No | eu_aspect | ++---+ | 270820 | X | | 192107 | S

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-09-20T12:08:38Z] Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717) (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 375544 merged by jenkins-bot: [operations/mediawiki-config@master] Enable statement usage tracking on elwiki https://gerrit.wikimedia.org/r/375544TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. I've just scheduled the deployment to elwiki for 12:00 UTC tomorrow (September 20).TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Bawolff, eranroz, Ottomata, PokestarFan,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

eranroz added a comment. I suggest to deploy it ( https://gerrit.wikimedia.org/r/#/c/375544/ ) in SWAT window on Monday/Tuesday (Morning SWAT): https://wikitech.wikimedia.org/wiki/Deployments#Monday.2C.C2.A0September.C2.A018TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Lydia_Pintscher added a comment. @hoo Do you have an estimate when you'll be able to do the deployment on elwiki? People are asking for this quite a bit now.TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

eranroz added a comment. In hewiki we started to use getBestStatements(Q,P) instead of loading the whole entity on 2/9 - so no X tracking (but O). See graphana: https://grafana.wikimedia.org/dashboard/db/wikidata-entity-usage-project?orgId=1=hewiki=1502573168310=1505165168311=15 and this is

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Bawolff added a comment. I suspect that having more rows in recentchanges is much much worse than having more rows in wbc_entity_usage.TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: BawolffCc: Bawolff,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

daniel added a comment. In T151717#3598520, @Bawolff wrote: My hope is that by using fine grained tracking on wbc_entity_usage, only when someone edits the actual property in use, will a wikidata edit be propogated to the local recentchanges table, which would significantly reduce the size of

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Bawolff added a comment. In T151717#3598222, @Halfak wrote: I think the idea is that we'll be able to include wbc_entity_usage to increase granularity in watchlists once this is solved for. It will require some new work though :) Happy to see @Bawolff excited about this functionality. My

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Halfak added a comment. I think the idea is that we'll be able to include wbc_entity_usage to increase granularity in watchlists once this is solved for. It will require some new work though :) Happy to see @Bawolff excited about this functionality.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

eranroz added a comment. In T151717#3595278, @Bawolff wrote: I suspect that fixing this bug will significantly help with T171027 (watchlists being too slow.). In particular, I would suggest prioritizing deploying to rowiki (Since its small enough to be good test wiki but is highly effected by

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. In T151717#3574731, @eranroz wrote: What is the current status of it? Sorry, seems this fell between the cracks again. I'll find a deployment date for this soon (this needs to be monitored).TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 375544 had a related patch set uploaded (by Eranroz; owner: Eranroz): [operations/mediawiki-config@master] Enable usage aspect C on elwiki https://gerrit.wikimedia.org/r/375544TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

eranroz added a comment. What is the current status of it? use elwiki_p; select eu_aspect, count(*) from wbc_entity_usage group by eu_aspect; +---+--+ | eu_aspect | count(*) | +---+--+ | L.el |97899 | | L.en | 6477 | | O |79429 | | S

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Halfak added a comment. One more thing. Part of our test deployment to elwiki as described in this task is to help get a sense for the rate that these usage rows will be written to MariaDB, so we're likely to have a better answer for you soon.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Halfak added a comment. Hard to say. N for each page rendering where N is the number of wikidata "usages" on the page. A lua template or parser function can use labels, descriptions, statements, sitelinks, or ... other... ??? TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Ottomata added a comment. I haven't fully grokked this ticket, but in general I am for events! :) 2 qs: How many events per second? What would be emitting this event? Mediawiki? TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Halfak added a comment. Talked to @hoo. If everything goes right with the deployment, we should be able to test this in the last week of Aug or first week of Sept. If we have a problem with the deployment, it will probably get pushed back ~2 weeks. We'll know more in 2-3 weeks.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment.Herald added a subscriber: PokestarFan. The functionality needed for this has been merged into master now. Once it's deployed (which should happen in 2-3 weeks), we can go ahead with the trial (which is a simple configuration change).TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Hall1467 added a comment. @Ladsgroup: I believe this is the only related open patch right now: https://gerrit.wikimedia.org/r/#/c/355104/. Is that right, @hoo?TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Ladsgroup added a comment. Just to clarify, Is gerrit:336542 is the only open patch right now?TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LadsgroupCc: Ladsgroup, Stashbot, gerritbot, Halfak, jcrespo,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-05-31T13:26:36Z] Synchronized wmf-config/Wikibase-production.php: WikibaseClient: Don't persist Statement usages (T151717) (duration: 00m 41s)TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 355101 merged by jenkins-bot: [operations/mediawiki-config@master] WikibaseClient: Don't persist Statement usages https://gerrit.wikimedia.org/r/355101TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Hall1467 added a comment. @hoo: elwiki sounds good to me.TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Hall1467Cc: gerritbot, Halfak, jcrespo, TomT0m, Hall1467, hoo, zhuyifei1999, Eloquence,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 355101 had a related patch set uploaded (by Hoo man; owner: Hoo man): [operations/mediawiki-config@master] WikibaseClient: Don't persist Statement usages https://gerrit.wikimedia.org/r/355101TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. I ran the numbers briefly and both kowiki and trwiki don't look as interesting to me. Therefore I would suggest going with elwiki, which has the following usages currently: ++---+ | count | eu_aspect | ++---+ | 246228 | X | | 179952 |

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 353154 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Add EntityUsage::STATEMENT_USAGE and adopt UsageAccumulator https://gerrit.wikimedia.org/r/353154TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

gerritbot added a comment. Change 353154 had a related patch set uploaded (by Hoo man; owner: Hoo man): [mediawiki/extensions/Wikibase@master] Add EntityUsage::STATEMENT_USAGE and adopt UsageAccumulator https://gerrit.wikimedia.org/r/353154TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. In T151717#3184079, @jcrespo wrote: We want to collect additional information on one of these wikis for a while If that doesn't involve a schema change, sure. It doesn't: We will just write additional rows into the existing table. Should we choose to not follow this path,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. We want to collect additional information on one of these wikis for a while If that doesn't involve a schema change, sure.TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc:

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

hoo added a comment. @jcrespo Just to make this super clear: We want to collect additional information on one of these wikis for a while (this can be turned of at any point in time). After that, we will asses the impact and come up with an estimate for other wikis. Depending on how this goes, we

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Hall1467 added a comment. @jcrespo: Related to your first comment, the patch that I provided a link to (in my comment from one month ago) is now invalid since we are no longer planning on using a separate table in order to implement statement tracking. A new patch should be ready soon implementing

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. To clarify, I have to be specially strict in this particular case because in the past, wbc_entity_usage (with the exception of linksupdate job) was a large point of contention and a major cause of lag, and this ticket starts by saying: we'd write a lot (?) more rows to

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. I did not understand your last comment, is the previous patch invalid? Do you have another patch to show me?TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Halfak,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Hall1467 added a comment. @jcrespo: We have decided to update the existing wbc_entity_usage table in order to allow for statement tracking (probably done via the "eu_aspect_id" field). We would do an initial deployment to a medium-sized Wikipedia at first and calculate database IO load. We will be

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Hall1467 added a comment. @jcrespo: Asking my question on here per your request. We were wondering what your thoughts are for the proposed logging table. See statement_usage.sql (https://gerrit.wikimedia.org/r/#/c/336542/19/client/sql/statement_usage.sql) for the schema. The schema is

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. To clarify, I am not saying it should be one way or another, what I am asking is: measure the write load impact Have into account both options, and be aware of them (e.g. maybe it is now worth it now, but we can prepare things so if it is needed in the future, we do not

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Halfak added a comment. Thanks @jcrespo. Re. IO how would we generate the kind of estimates you would need in order to sign off on this type of change? Re. different set of servers that would work for some of the immediate use-cases, but I'm not sure it would work for the other use cases. I

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

jcrespo added a comment. Storage is not a problem. I wonder what is the impact in IO activity (write QPS). Could we separate usage tracking to a different set of servers? This table(s) are probably very dynamic, but also probably not 100% in sync with the content edits (handled on asynchronous

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Hall1467 added a comment. To follow up on @Halfak's database usage assessments, the estimate of 5 properties per entity/page relationship seems reasonable and conservative since the average number of statements per entity is in fact ~5 as seen here:

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Halfak added a comment. @daniel the storage savings are roughly the same even if we use a varbinary property ID. Normalization is what gets us most of the gain. Could you be more specific about what pattern you are referring to from that RFC?TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Halfak added a comment. @daniel, it seems that normalizing the property references out of that table would save considerable amounts of space. E.g. with the current scheme, we have eu_row_id -- bigint(20) -- 8 bytes eu_entity_id -- varbinary(255) -- 256 bytes eu_aspect_id -- varbinary(37) -- 38

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

daniel added a comment. @Hall1467 we already have a table for logging this kind of thing: wbc_entity_usage. We'd just be putting a lot more data in it. Introducing a specialized table would be possible, but a lot more work, and probably not worth the pain.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Hall1467 added a comment. If we choose to go the database route instead of EventLogging, my initial thoughts would be to create a new table containing (modeled after @Halfak's reference and the current entity usage documentation:

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

daniel added a comment. @Hall1467 If you want to work on it, sure, go ahead! However, note that we may choose to not track this information if it turns out that the potential benefit is not worth the large amount of extra entries in the database. There is no decision on that yet, and I'm not sure

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Hall1467 added a comment. I'd be happy to take the lead on the implementation of this task. Would that be okay?TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Hall1467Cc: TomT0m, Hall1467, hoo, zhuyifei1999,

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

Hall1467 added a comment. I think a good way to go about this tracking would be by overriding the Lua direct access method for the table that represents the entity. This can be done via an “__index” metamethod that would log statement usage (see this example of tracking: