[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed
ori added a comment. Follow-up items to get the Puppet repo on deployment-puppetmaster04 in good shape: - The two cherry-picked reverts should be removed (https://gerrit.wikimedia.org/r/c/operations/puppet/+/823638, https://gerrit.wikimedia.org/r/c/operations/puppet/+/823639) and the changes they revert should be updated to not be incompatible with the Beta Cluster. - https://gerrit.wikimedia.org/r/c/operations/puppet/+/668701 needs to be rebased and merged or re-cherry-picked on deployment-puppetmaster04. To get it to apply I had to manually resolve a conflict, and I'm not sure I did it correctly. So the actual diff on deployment-puppetmaster04 is not consistent with what's on Gerrit. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: thcipriani, ori, Zabe, ArielGlenn, bking, RhinosF1, Ryasmeen, matmarex, ppelberg, Daimona, TheresNoTime, Aklapper, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, TheReadOnly, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, CptViraj, Kent7301, joker88john, DannyS712, CucyNoiD, Nandana, NebulousIris, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Liudvikas, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Ladsgroup, Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Nandana, Jony, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori added a comment. The change I'm proposing (I6b36c1647 <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/674472/>) should not introduce any significant discontinuity, since the counter delta is proportional to the sampling rate: i.e., with a sampling rate of 1%, 1:100 incrementStatsKey() calls will be processed, but each one will increment the counter by 100. TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Ladsgroup, Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Jony, lucamauri, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori added a comment. By the way: this would be a good first patch for a new contributor, and I'd be happy to mentor someone through it. TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Nandana, Jony, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori added a comment. ...and if you increment the stat by a 100 each time, you wouldn't need to change the dashboards. TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Nandana, Jony, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori added a comment. I like this idea, but wouldn't N reset to zero every time we called into Lua code? How would you persist it? An alternative approach would be random sampling: e.g., log only if `math.random(100) == 1`. TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Nandana, Jony, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T147798: Wikibase repo UI is sometimes not properly starting up (on Firefox)
ori added a comment. Can you add some ad hoc console#log statements to your local instance?TASK DETAILhttps://phabricator.wikimedia.org/T147798EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: oriCc: ori, Aklapper, Krinkle, aude, thiemowmde, Jonas, Lydia_Pintscher, Tobi_WMDE_SW, hoo, Vali.matei, D3r1ck01, Izno, Wikidata-bugs, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. We are definitely sending duplicate purges. I added some debug logging and saw URLs get purged a half dozen times for the same job. TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: EBernhardson, Smalyshev, gerritbot, Legoktm, Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, aaron, faidon, Joe, ori, BBlack, Aklapper, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a subscriber: EBernhardson. ori added a comment. @EBernhardson made it so when a job fragments into a number of child jobs, each child job has the same request ID as its parent. This also makes it possible to aggregate PURGEs by individual parent job: Top PURGE issuers by orig. request id - | Orig. request id | Percent | | | --- | | V07acApAEK4AAA-p8t4AAACK | 24.43% | | V09G2wpAIDEAAAoe07kAAABE | 10.05% | | V0xKtwpAEK8AAFSGAicAAACI | 3.59% | | V0xKtQpAEIEAAAvjcJIY | 3.45% | | V08jWwpAEK4AAHA7yEsV | 3.44% | | V08P2QpAADUAABlSU9sB | 2.49% | | V08o4QpAICwAABlKSrUAAACJ | 2.39% | | fe4a69ca01a1c0e76f782cd9| 2.34% | | 73fd09f28d554f56484bed95| 1.97% | | ceeea6f37cefae6a4a86478b| 1.90% | This tells us that of the 10,718,138 PURGEs issued during the observation period, approximately 2,618,441 PURGEs were issued by a single htmlCacheUpdate job for svwiki. svwiki only has 6,464,684 pages in total (of which 3,199,790 are content pages), so that means nearly half of svwiki was purged over the course of four hours. My hunch is that purges are not getting correctly de-duplicated, so a single page can get purged multiple times by children of a single root job. I wonder whether we correctly de-duplicate the case when a template is indirectly included on a page via two or more different intermediate templates. For example, on enwiki, [[Template:Navbox]] is included in [[Barack Obama]] both through [[Template:US Presidents]] and [[Template:Current U.S. Governors]]. TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: EBernhardson, Smalyshev, gerritbot, Legoktm, Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, aaron, faidon, Joe, ori, BBlack, Aklapper, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. Breakdown of 10,718,138 PURGEs, captured on 2016-06-01 between 18:00 and 22:00 UTC: Top PURGE issuers by wiki - | Wiki | Percent | | | --- | | svwiki | 24.64% | | itwiki | 12.05% | | enwiki | 11.46% | | frwiki | 9.44% | | enwiktionary | 8.00% | | dewiki | 4.91% | | arwiki | 4.66% | | commonswiki | 4.58% | | cawiki | 2.54% | | shwiki | 1.81% | Top PURGE issuers by host - | Host | Percent | | -- | --- | | mw1011 | 7.85% | | mw1163 | 5.95% | | mw1165 | 5.75% | | mw1013 | 5.74% | | mw1167 | 5.73% | | mw1003 | 5.71% | | mw1169 | 5.68% | | mw1005 | 5.66% | | mw1015 | 5.66% | | mw1001 | 5.64% | Top PURGE issuers by entrypoint --- | Entrypoint | Percent | | | --- | | RunJobs.php | 91.48% | | api.php | 5.05% | | index.php| 3.46% | | MWScript.php | 0.01% | Top PURGE issuers by user (anonymized) -- | User (anonymized) | Percent | | - | --- | | 127.0.0.1 | 91.57% | || 0.88% | || 0.49% | || 0.42% | || 0.33% | || 0.20% | || 0.18% | || 0.16% | || 0.13% | || 0.10% | Top PURGE issuers by job type - | Job type | Percent | | -- | --- | | htmlCacheUpdate| 78.82% | | ChangeNotification | 11.98% | | (not a job)| 8.52% | | LocalGlobalUserPageCacheUpdateJob | 0.42% | | MassMessageJob | 0.13% | | flaggedrevs_CacheUpdate| 0.10% | | cirrusSearchLinksUpdatePrioritized | 0.01% | | refreshLinksPrioritized| 0.01% | | UpdateRepoOnMove | 0.01% | | LocalPageMoveJob | 0.01% | Top PURGE issuers by job + wiki pair | Job + wiki pair| Percent | | -- | --- | | htmlCacheUpdate / svwiki | 23.70% | | htmlCacheUpdate / itwiki | 10.76% | | htmlCacheUpdate / enwiki | 9.77% | | htmlCacheUpdate / enwiktionary | 7.97% | | htmlCacheUpdate / frwiki | 6.08% | | htmlCacheUpdate / arwiki | 4.34% | | htmlCacheUpdate / commonswiki | 3.87% | | ChangeNotification / frwiki| 3.09% | | ChangeNotification / dewiki| 2.81% | | htmlCacheUpdate / cawiki | 2.50% | TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Smalyshev, gerritbot, Legoktm, Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, aaron, faidon, Joe, ori, BBlack, Aklapper, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reopened] T103429: Investigation: Parser save hook handler does master writes in GETs
ori reopened this task as "Open". ori added a comment. Doing this via the job queue is not a good solution, in my opinion. There is something fundamentally broken about tying updates to parser cache events. Subscribing to changes via a ParserCacheSaveComplete hook handler is a bit like a department store updating its inventory any time a customer touches an item: sure, it covers all the relevant transactions, but it also creates a large amount of unnecessary work, because customers may handle an item for a number of reasons (to try it on or read the label), not all of which result in a transaction. ParserCacheSaveComplete events are the same way: they cover all the cases in which an edit is made, but they also fire in cases which do not involve a transaction. For example, the mobile and desktop web sites use different key-spaces to avoid polluting each other's parser cache entries with platform-specific artifacts. Hooking into ParserCacheSaveComplete is the wrong thing to do, because (AIUI) it isn't the really the parser cache that Wikibase cares about, and because it muddles the distinction between read-only and read/write requests. For example, RefreshLinks jobs may opportunistically refresh the parser cache <https://github.com/wikimedia/mediawiki/blob/6e9b4f0/includes/jobqueue/jobs/RefreshLinksJob.php#L187-L195>. The fact that Wikibase enqueues //another// job on ParserCacheSaveComplete makes it harder to reason about the job queue. Finally, like the example above, it creates a //large// amount of unnecessary work: when I commented out the hook for a few minutes yesterday (to investigate https://phabricator.wikimedia.org/T129517), I saw a marked drop in memcached traffic. TASK DETAIL https://phabricator.wikimedia.org/T103429 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, gerritbot, hoo, Addshore, Tobi_WMDE_SW, daniel, Lydia_Pintscher, Aklapper, aaron, D3r1ck01, Izno, Wikidata-bugs, aude, GWicke, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Merged] T103346: phpunit hhvm failure: LuaSandbox: TextLibraryTests[87]: json decode, invalid values (trailing comma)
ori closed this task as a duplicate of T128029: TemplateData's PHP JSON validation isn't strict enough because WMF cluster's HHVM allows trailing commas. TASK DETAIL https://phabricator.wikimedia.org/T103346 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mattflaschen, ori Cc: Catrope, Mattflaschen, gerritbot, StudiesWorld, daniel, Anomie, JanZerebecki, Aklapper, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, Mbch331, Krenair, Joe, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. In https://phabricator.wikimedia.org/T124418#1986594, @BBlack wrote: > Regardless, the average rate of HTCP these days is normally-flat-ish (a few > scary spikes aside), and is mostly throttled by the jobqueue. The question > still remains: what caused permanent, large bumps in the jobqueue > htmlCacheUpdate insertion rate on ~Dec4, ~Dec11, and ~Jan20? I think it was rMWd03c5be5e340: Make HTMLCacheUpdate always use the job queue <https://phabricator.wikimedia.org/rMWd03c5be5e3400b43317b18fcde16d86209b35014>. I am not sure how I missed that change until now. The change made all purging go through the job queue, whereas previously if the number of backlinks was less than 100, the purges would be dispatched synchronously, in the main request context. TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Legoktm, Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, Denniss, aaron, faidon, Joe, ori, BBlack, Aklapper, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. Distribution of (indirect) callers of `HTMLCacheUpdate::__construct` for the past 20 minutes: [fluorine:/a/mw-log] $ python /home/ori/cacheUpdateGrepper.py | dist Key|Ct (Pct) Histogram Wikibase\Repo\Api\ModifyEntity->attemptSaveEntity|6764 (27.07%) ██ SubmitAction->show|6311 (25.26%) █▋ ApiMain->executeActionWithErrorHandling|6291 (25.18%) █▌ Wikibase\Repo\Api\ModifyClaim->attemptSaveEntity| 940 (3.76%) █▎ ApiMain->execute| 803 (3.21%) ███▉ MediaWiki->performAction| 579 (2.32%) █▊ LoadBalancer->commitMasterChanges| 392 (1.57%) ███▉ EditAction->show| 331 (1.32%) ███▎ ApiMain->executeAction| 197 (0.79%) ██ FormlessAction->show| 150 (0.60%) █▌ Wikibase\Repo\Interactors\ItemMergeInteractor->mergeItems| 120 (0.48%) █▎ Wikibase\Repo\UpdateRepo\UpdateRepoJob->run| 106 (0.42%) █ TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, Denniss, aaron, faidon, Joe, ori, BBlack, Aklapper, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. Distribution of purge URLs by hostname: [fluorine:/a/mw-log] $ field 7 AdHocDebug.log | sed 's/\.m\./\./' | sort | dist | head Key|Ct (Pct)Histogram en.wiktionary.org|893075 (18.66%) █ fr.wikipedia.org|836392 (17.48%) ██▋ commons.wikimedia.org|758195 (15.84%) █▊ en.wikipedia.org|435543 (9.10%) █▍ it.wikipedia.org|260668 (5.45%) █▌ zh.wikipedia.org|184594 (3.86%) █ ar.wikipedia.org|174000 (3.64%) ███▊ ko.wiktionary.org|148310 (3.10%) ▉ ko.wikipedia.org| 76192 (1.59%) ▋ th.wikipedia.org| 72084 (1.51%) ▎ TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, Denniss, aaron, faidon, Joe, ori, BBlack, Aklapper, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. In https://phabricator.wikimedia.org/T124418#1963710, @JanZerebecki wrote: > https://grafana-admin.wikimedia.org/dashboard/db/tmp-t124418 What is that dashboard supposed to be showing? It is very hard to make sense of. TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, Denniss, aaron, faidon, Joe, ori, BBlack, Aklapper, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori added a comment. https://phabricator.wikimedia.org/tag/wikidata/ folks (especially @daniel and @Lydia_Pintscher): Just a quick note to apologize for my conduct on this task. I think I was quick to escalate things, and I did it in a manner that probably only succeeded at raising the stress level of everyone involved. I'll treat it as a learning experience. :) TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo, ori Cc: Tobi_WMDE_SW, ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori added a blocking task: T48643: [Story] Dispatching via delayed jobs (instead of cron script). TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T48643: [Story] Dispatching via delayed jobs (instead of cron script)
ori added a blocked task: T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections. TASK DETAIL https://phabricator.wikimedia.org/T48643 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, Tobi_WMDE_SW, JanZerebecki, Wikidata-bugs, Abraham, Nemo_bis, Denny, aude, Ricordisamoa, Lydia_Pintscher, daniel, hoo, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori added a comment. In https://phabricator.wikimedia.org/T118162#1795369, @jcrespo wrote: > - The cron jobs run as wikiadmin, which are on purpose not limited in > execution time, so there is no protection against them overflowing a database > server In https://phabricator.wikimedia.org/T118162#1795406, @daniel wrote: > - rewriting the dispatch logic to rely on the job queue has long been on the > todo list, see T48643: [Story] Dispatching via delayed jobs (instead of cron > script) <https://phabricator.wikimedia.org/T48643>. It's not trivial to get > right, though. Ok, so it sounds like we have: - A very severe problem. - Agreement on how to mitigate it. @Lydia_Pintscher, could you please make this a priority? Especially since the sprint goal of checking if this has been resolved has just been accomplished for you by Jaime. TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori added subscribers: mobrovac, ori. ori added a comment. @daniel, could you please make sure to sync up with the Services team to see how https://phabricator.wikimedia.org/T114443 could lay the groundwork for solving this problem better? @mobrovac, could you please act as liaison / point-person? TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T116568: navigation timing stats specific to wikidata.org item? pages
ori removed a project: Performance-Team. TASK DETAIL https://phabricator.wikimedia.org/T116568 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JanZerebecki, ori Cc: Krinkle, gerritbot, thiemowmde, adrianheine, Addshore, JanZerebecki, Lydia_Pintscher, Aklapper, Ricordisamoa, Wikidata-bugs, aude ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T117398: number of database updates multiplied x3 since 29 November
ori added a subscriber: Performance-Team. ori set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T117398 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Performance-Team, Aklapper, jcrespo, Wikidata-bugs, aude, GWicke, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T114995: [Bug] Varnish cache invalidation not working for m.wikidata.org
ori closed this task as "Resolved". ori claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T114995 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: gerritbot, faidon, Addshore, BBlack, Jdlrobson, daniel, Lydia_Pintscher, Aklapper, hoo, Wikidata-bugs, aude, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP
ori added a comment. In https://phabricator.wikimedia.org/T114443#1731284, @GWicke wrote: > See https://phabricator.wikimedia.org/T88459#1604768. tl;dr: It's not > necessarily clear that saving very little code (see above) for EL schema > fetching outweights the cost of additional hardware. Could you explain how you arrived at the figure of 50k requests per second, which you project for this service? TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, ori Cc: mark, MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, daniel, JanZerebecki, RobLa-WMF, Jay8g, fgiunchedi, Dzahn, jeremyb, Legoktm, chasemp, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP
ori added a comment. In https://phabricator.wikimedia.org/T114443#1701193, @GWicke wrote: > @Nuria, see the task description, heading "Initial use cases". Potential applications are one thing; a concise problem-statement another. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, mark, JanZerebecki, RobLa-WMF, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T113665: SitesModuleWorker::getSitesHash() is slow
ori edited projects, added Performance; removed Performance-Team. TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: adrianheine, Krenair, MaxSem, JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs, GWicke ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T113665: SitesModuleWorker::getSitesHash() is slow
ori changed the title from "SitesModuleWorker::getSitesHash()" to "SitesModuleWorker::getSitesHash() is slow". ori set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Krenair, MaxSem, JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T113665: SitesModuleWorker::getSitesHash()
ori added a comment. SiteList (why can't it just be an array?) extends GenericArrayObject which extends ArrayObject. ArrayObject provides a `ksort()` method which would be useful here, except that SiteList maintains its own mappings of offsets to values in internal arrays. TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Krenair, MaxSem, JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T113665: SitesModuleWorker::getSitesHash()
ori added a subscriber: MaxSem. ori added a comment. @MaxSem looked at this code and instantly noticed that it is repeatedly sorting the cached value whenever it is retrieved (actually, whenever the method is called), and remarked that it would be far more sensible to do this once, when //saving// the value. TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: MaxSem, JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T113665: SitesModuleWorker::getSitesHash()
ori created this task. ori added subscribers: ori, aude, hoo, daniel, JeroenDeDauw. ori added projects: Performance-Team, Wikidata. Herald added a subscriber: Aklapper. TASK DESCRIPTION Looking at performance data for `load.php` requests, I notice that we spend quite a lot of time in `SitesModuleWorker::getSitesHash()`. So I would like to know what that method does. I grep for it, and find that it calls `SitesModuleWorker::getSites()`. `SitesModuleWorker::getSites()`, in turn, calls `SiteStore::getSites()`. But SiteStore is an interface, and I don't know which concrete implementation SitesModuleWorker is using, because the implementation is specified via a parameter to `SitesModuleWorker::__construct()`. So I need to look and see where SitesModuleWorker is instantiated. That takes me to `lib/includes/modules/SitesModule.php:29`, where I see that it is `SiteSQLStore::newInstance()`. OK. But where is SiteSQLStore? Oh, I see that it's in Core. OK, let's pull it up and see what its `SiteSQLStore::getSites()` method does. Looks like it doesn't have one! SiteSQLStore extends CachingSiteStore, so I guess the method is implemented there. OK, let's go look at `CachingSiteStore::getSites()`. OK, so CachingSiteStore tries to get the sites from its cache object, which is a BagOStuff. I don't know which, because, again, that is up to the caller of `CachingSiteStore::__construct()`. Which is, it turns out, `SiteSQLStore::newInstance()`. But `SiteSQLStore::newInstance()` itself takes a `$cache` parameter. Fortunately, I remember that the caller is back in `SitesModule.php:29`, and I see that it does not specify a value for `$cache`, which means that it must be defaulting to `CACHE_ACCEL` for HHVM. Now I know what happens on a cache hit. Progress. But what about a cache miss? In that case, `CachingSiteStore::getSites()` calls `$this->siteStore->getSites();`. But what is its `$siteStore`? That is up to the caller of `CachingSiteStore::__construct()`. So I go back to `SiteSQLStore::newInstance()`, and find out that it is passing in a `DBSiteStore` instance. OK, let's go to DBSiteStore. `DBSiteStore::getSites()` calls `$this->loadSites();`. That leads me to `DBSiteStore::loadSites()`. Here, finally, I see an actual database query. All in all, to figure out what `SitesModuleWorker::getSitesHash()` is doing, I had to examine eight methods on no fewer than **six classes**. I still don't know why it is slow: cache hits and misses are not instrumented, and I haven't analyzed the query to see how the database executes it, but because I spent half an hour opening and closing files I am too exasperated to do that work. I have repeatedly suggested using a static array in `wmf-config` for this data, and at one point I believe this may have actually been implemented. Why are we not doing that? TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Raised Priority] T111304: Fatal error: Call to undefined method OOUI\Element::serializeForApiResult() in /srv/mediawiki/php-1.26wmf21/vendor/oojs/oojs-ui/php/Element.php o
ori added subscribers: greg, ori. ori raised the priority of this task from "Normal" to "High". ori added a comment. This is *still* causing fatals in production. Can somebody fix this, please? @greg, could you please look after this? Anomie's suggestion from https://phabricator.wikimedia.org/T111304#1606372 is sound. TASK DETAIL https://phabricator.wikimedia.org/T111304 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, greg, Jdlrobson, Anomie, Legoktm, Krenair, Jdforrester-WMF, Aklapper, mmodell, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Lydia_Pintscher, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T58602: avoid fetching SiteList object from memcached
ori added a comment. Thanks for the quick response. TASK DETAIL https://phabricator.wikimedia.org/T58602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aude, ori Cc: gerritbot, Aklapper, daniel, Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride, csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, Unknown Object (MLST), GWicke, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T58602: avoid fetching SiteList object from memcached
ori added a comment. In https://phabricator.wikimedia.org/T58602#1463722, @hoo wrote: > @aude is going to poke at this when she's back from Wikimania. Cool, thanks. TASK DETAIL https://phabricator.wikimedia.org/T58602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aude, ori Cc: Aklapper, daniel, Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride, csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, Unknown Object (MLST), GWicke, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Raised Priority] T58602: avoid fetching SiteList object from memcached
ori raised the priority of this task from "High" to "Unbreak Now!". ori added a comment. Herald added a subscriber: Aklapper. Every so often I crunch some statistics about memcached usage. I can't tell you how demoralizing it is to find `sites/SiteList#2014-03-17+Site:2013-01-23` near the top again and again. If it's still there the next time I check I'm going to start disabling extensions. Changing priority to UBN! for that reason. Please fix this by making the list a static array in a PHP file in `operations/mediawiki-config` and then include it in `CommonSettings.php`. This way HHVM will compile it to byte-code and the OS will keep it in memory. TASK DETAIL https://phabricator.wikimedia.org/T58602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, daniel, Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride, csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, Unknown Object (MLST), GWicke, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T88550: Mourn Titan
ori added a subscriber: ori. ori added a comment. In https://phabricator.wikimedia.org/T88550#1395307, @MZMcBride wrote: > What replaced Titan? Blazegraph <http://www.blazegraph.com/> TASK DETAIL https://phabricator.wikimedia.org/T88550 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: MaxSem, ori Cc: ori, MZMcBride, Krenair, Smalyshev, Legoktm, MaxSem, JanZerebecki, Aklapper, Manybubbles, jkroll, Wikidata-bugs, Jdouglas, aude ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reopened] T58602: avoid fetching SiteList object from memcached
ori reopened this task as "Open".
ori added a comment.
`{$wgDBname}:SiteList:sites/SiteList#2014-03-17+Site:2013-01-23` items are at
or near the top of memcached keys sorted by bandwidth utilization on the
production memcached cluster. This really needs to be fixed and stay fixed.
TASK DETAIL
https://phabricator.wikimedia.org/T58602
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Wikidata-bugs, ori
Cc: Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride,
csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, GWicke, wikidata-bugs
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Block] T76705: SiteStore / SiteList performance and caching (tracking)
ori reopened blocking task T58602: avoid fetching SiteList object from memcached as "Open". TASK DETAIL https://phabricator.wikimedia.org/T76705 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, daniel, Wikidata-bugs, aude, GWicke, wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Block] T76705: SiteStore / SiteList performance and caching (tracking)
ori reopened blocking task T58602: avoid fetching SiteList object from memcached as "Open". TASK DETAIL https://phabricator.wikimedia.org/T76705 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, daniel, Wikidata-bugs, aude, GWicke, wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reopened] T58602: avoid fetching SiteList object from memcached
ori reopened this task as "Open".
ori added a comment.
`{$wgDBname}:SiteList:sites/SiteList#2014-03-17+Site:2013-01-23` items are at
or near the top of memcached keys sorted by bandwidth utilization on the
production memcached cluster. This really needs to be fixed and stay fixed.
TASK DETAIL
https://phabricator.wikimedia.org/T58602
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Wikidata-bugs, ori
Cc: Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride,
csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, GWicke, wikidata-bugs
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T86305: ApiStashEdit silently loses custom DataUpdates.
ori added a comment. @daniel, thanks for clarifying. Could you also explain (in practical terms) what are the updates being performed? TASK DETAIL https://phabricator.wikimedia.org/T86305 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign . EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, daniel, aude, hoo, Lydia_Pintscher, aaron, ori, Jackmcbarn, Anomie, cscott, Wikidata-bugs, Jdforrester-WMF, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T86305: ApiStashEdit silently loses custom DataUpdates.
ori added a comment. @daniel Can you explain why this work needs to be done synchronously, blocking the user? TASK DETAIL https://phabricator.wikimedia.org/T86305 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign . EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, daniel, aude, hoo, Lydia_Pintscher, aaron, ori, Jackmcbarn, Anomie, cscott, Wikidata-bugs, Jdforrester-WMF, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
