[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed
ori added a comment. Follow-up items to get the Puppet repo on deployment-puppetmaster04 in good shape: - The two cherry-picked reverts should be removed (https://gerrit.wikimedia.org/r/c/operations/puppet/+/823638, https://gerrit.wikimedia.org/r/c/operations/puppet/+/823639) and the changes they revert should be updated to not be incompatible with the Beta Cluster. - https://gerrit.wikimedia.org/r/c/operations/puppet/+/668701 needs to be rebased and merged or re-cherry-picked on deployment-puppetmaster04. To get it to apply I had to manually resolve a conflict, and I'm not sure I did it correctly. So the actual diff on deployment-puppetmaster04 is not consistent with what's on Gerrit. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: thcipriani, ori, Zabe, ArielGlenn, bking, RhinosF1, Ryasmeen, matmarex, ppelberg, Daimona, TheresNoTime, Aklapper, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, TheReadOnly, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, CptViraj, Kent7301, joker88john, DannyS712, CucyNoiD, Nandana, NebulousIris, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Liudvikas, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Ladsgroup, Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Nandana, Jony, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori added a comment. The change I'm proposing (I6b36c1647 <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/674472/>) should not introduce any significant discontinuity, since the counter delta is proportional to the sampling rate: i.e., with a sampling rate of 1%, 1:100 incrementStatsKey() calls will be processed, but each one will increment the counter by 100. TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Ladsgroup, Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Jony, lucamauri, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori added a comment. By the way: this would be a good first patch for a new contributor, and I'd be happy to mentor someone through it. TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Nandana, Jony, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori added a comment. ...and if you increment the stat by a 100 each time, you wouldn't need to change the dashboards. TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Nandana, Jony, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T277817: incrementStatsKey() calls in Wikibase Lua are expensive
ori added a comment. I like this idea, but wouldn't N reset to zero every time we called into Lua code? How would you persist it? An alternative approach would be random sampling: e.g., log only if `math.random(100) == 1`. TASK DETAIL https://phabricator.wikimedia.org/T277817 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Lydia_Pintscher, hoo, Aklapper, ori, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Nandana, Jony, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T147798: Wikibase repo UI is sometimes not properly starting up (on Firefox)
ori added a comment. Can you add some ad hoc console#log statements to your local instance?TASK DETAILhttps://phabricator.wikimedia.org/T147798EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: oriCc: ori, Aklapper, Krinkle, aude, thiemowmde, Jonas, Lydia_Pintscher, Tobi_WMDE_SW, hoo, Vali.matei, D3r1ck01, Izno, Wikidata-bugs, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. We are definitely sending duplicate purges. I added some debug logging and saw URLs get purged a half dozen times for the same job. TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: EBernhardson, Smalyshev, gerritbot, Legoktm, Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, aaron, faidon, Joe, ori, BBlack, Aklapper, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a subscriber: EBernhardson. ori added a comment. @EBernhardson made it so when a job fragments into a number of child jobs, each child job has the same request ID as its parent. This also makes it possible to aggregate PURGEs by individual parent job: Top PURGE issuers by orig. request id - | Orig. request id | Percent | | | --- | | V07acApAEK4AAA-p8t4AAACK | 24.43% | | V09G2wpAIDEAAAoe07kAAABE | 10.05% | | V0xKtwpAEK8AAFSGAicAAACI | 3.59% | | V0xKtQpAEIEAAAvjcJIY | 3.45% | | V08jWwpAEK4AAHA7yEsV | 3.44% | | V08P2QpAADUAABlSU9sB | 2.49% | | V08o4QpAICwAABlKSrUAAACJ | 2.39% | | fe4a69ca01a1c0e76f782cd9| 2.34% | | 73fd09f28d554f56484bed95| 1.97% | | ceeea6f37cefae6a4a86478b| 1.90% | This tells us that of the 10,718,138 PURGEs issued during the observation period, approximately 2,618,441 PURGEs were issued by a single htmlCacheUpdate job for svwiki. svwiki only has 6,464,684 pages in total (of which 3,199,790 are content pages), so that means nearly half of svwiki was purged over the course of four hours. My hunch is that purges are not getting correctly de-duplicated, so a single page can get purged multiple times by children of a single root job. I wonder whether we correctly de-duplicate the case when a template is indirectly included on a page via two or more different intermediate templates. For example, on enwiki, [[Template:Navbox]] is included in [[Barack Obama]] both through [[Template:US Presidents]] and [[Template:Current U.S. Governors]]. TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: EBernhardson, Smalyshev, gerritbot, Legoktm, Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, aaron, faidon, Joe, ori, BBlack, Aklapper, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. Breakdown of 10,718,138 PURGEs, captured on 2016-06-01 between 18:00 and 22:00 UTC: Top PURGE issuers by wiki - | Wiki | Percent | | | --- | | svwiki | 24.64% | | itwiki | 12.05% | | enwiki | 11.46% | | frwiki | 9.44% | | enwiktionary | 8.00% | | dewiki | 4.91% | | arwiki | 4.66% | | commonswiki | 4.58% | | cawiki | 2.54% | | shwiki | 1.81% | Top PURGE issuers by host - | Host | Percent | | -- | --- | | mw1011 | 7.85% | | mw1163 | 5.95% | | mw1165 | 5.75% | | mw1013 | 5.74% | | mw1167 | 5.73% | | mw1003 | 5.71% | | mw1169 | 5.68% | | mw1005 | 5.66% | | mw1015 | 5.66% | | mw1001 | 5.64% | Top PURGE issuers by entrypoint --- | Entrypoint | Percent | | | --- | | RunJobs.php | 91.48% | | api.php | 5.05% | | index.php| 3.46% | | MWScript.php | 0.01% | Top PURGE issuers by user (anonymized) -- | User (anonymized) | Percent | | - | --- | | 127.0.0.1 | 91.57% | || 0.88% | || 0.49% | || 0.42% | || 0.33% | || 0.20% | || 0.18% | || 0.16% | || 0.13% | || 0.10% | Top PURGE issuers by job type - | Job type | Percent | | -- | --- | | htmlCacheUpdate| 78.82% | | ChangeNotification | 11.98% | | (not a job)| 8.52% | | LocalGlobalUserPageCacheUpdateJob | 0.42% | | MassMessageJob | 0.13% | | flaggedrevs_CacheUpdate| 0.10% | | cirrusSearchLinksUpdatePrioritized | 0.01% | | refreshLinksPrioritized| 0.01% | | UpdateRepoOnMove | 0.01% | | LocalPageMoveJob | 0.01% | Top PURGE issuers by job + wiki pair | Job + wiki pair| Percent | | -- | --- | | htmlCacheUpdate / svwiki | 23.70% | | htmlCacheUpdate / itwiki | 10.76% | | htmlCacheUpdate / enwiki | 9.77% | | htmlCacheUpdate / enwiktionary | 7.97% | | htmlCacheUpdate / frwiki | 6.08% | | htmlCacheUpdate / arwiki | 4.34% | | htmlCacheUpdate / commonswiki | 3.87% | | ChangeNotification / frwiki| 3.09% | | ChangeNotification / dewiki| 2.81% | | htmlCacheUpdate / cawiki | 2.50% | TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Smalyshev, gerritbot, Legoktm, Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, aaron, faidon, Joe, ori, BBlack, Aklapper, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reopened] T103429: Investigation: Parser save hook handler does master writes in GETs
ori reopened this task as "Open". ori added a comment. Doing this via the job queue is not a good solution, in my opinion. There is something fundamentally broken about tying updates to parser cache events. Subscribing to changes via a ParserCacheSaveComplete hook handler is a bit like a department store updating its inventory any time a customer touches an item: sure, it covers all the relevant transactions, but it also creates a large amount of unnecessary work, because customers may handle an item for a number of reasons (to try it on or read the label), not all of which result in a transaction. ParserCacheSaveComplete events are the same way: they cover all the cases in which an edit is made, but they also fire in cases which do not involve a transaction. For example, the mobile and desktop web sites use different key-spaces to avoid polluting each other's parser cache entries with platform-specific artifacts. Hooking into ParserCacheSaveComplete is the wrong thing to do, because (AIUI) it isn't the really the parser cache that Wikibase cares about, and because it muddles the distinction between read-only and read/write requests. For example, RefreshLinks jobs may opportunistically refresh the parser cache <https://github.com/wikimedia/mediawiki/blob/6e9b4f0/includes/jobqueue/jobs/RefreshLinksJob.php#L187-L195>. The fact that Wikibase enqueues //another// job on ParserCacheSaveComplete makes it harder to reason about the job queue. Finally, like the example above, it creates a //large// amount of unnecessary work: when I commented out the hook for a few minutes yesterday (to investigate https://phabricator.wikimedia.org/T129517), I saw a marked drop in memcached traffic. TASK DETAIL https://phabricator.wikimedia.org/T103429 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, gerritbot, hoo, Addshore, Tobi_WMDE_SW, daniel, Lydia_Pintscher, Aklapper, aaron, D3r1ck01, Izno, Wikidata-bugs, aude, GWicke, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Merged] T103346: phpunit hhvm failure: LuaSandbox: TextLibraryTests[87]: json decode, invalid values (trailing comma)
ori closed this task as a duplicate of T128029: TemplateData's PHP JSON validation isn't strict enough because WMF cluster's HHVM allows trailing commas. TASK DETAIL https://phabricator.wikimedia.org/T103346 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mattflaschen, ori Cc: Catrope, Mattflaschen, gerritbot, StudiesWorld, daniel, Anomie, JanZerebecki, Aklapper, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, Mbch331, Krenair, Joe, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. In https://phabricator.wikimedia.org/T124418#1986594, @BBlack wrote: > Regardless, the average rate of HTCP these days is normally-flat-ish (a few > scary spikes aside), and is mostly throttled by the jobqueue. The question > still remains: what caused permanent, large bumps in the jobqueue > htmlCacheUpdate insertion rate on ~Dec4, ~Dec11, and ~Jan20? I think it was rMWd03c5be5e340: Make HTMLCacheUpdate always use the job queue <https://phabricator.wikimedia.org/rMWd03c5be5e3400b43317b18fcde16d86209b35014>. I am not sure how I missed that change until now. The change made all purging go through the job queue, whereas previously if the number of backlinks was less than 100, the purges would be dispatched synchronously, in the main request context. TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Legoktm, Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, Denniss, aaron, faidon, Joe, ori, BBlack, Aklapper, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. Distribution of purge URLs by hostname: [fluorine:/a/mw-log] $ field 7 AdHocDebug.log | sed 's/\.m\./\./' | sort | dist | head Key|Ct (Pct)Histogram en.wiktionary.org|893075 (18.66%) █ fr.wikipedia.org|836392 (17.48%) ██▋ commons.wikimedia.org|758195 (15.84%) █▊ en.wikipedia.org|435543 (9.10%) █▍ it.wikipedia.org|260668 (5.45%) █▌ zh.wikipedia.org|184594 (3.86%) █ ar.wikipedia.org|174000 (3.64%) ███▊ ko.wiktionary.org|148310 (3.10%) ▉ ko.wikipedia.org| 76192 (1.59%) ▋ th.wikipedia.org| 72084 (1.51%) ▎ TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, Denniss, aaron, faidon, Joe, ori, BBlack, Aklapper, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. Distribution of (indirect) callers of `HTMLCacheUpdate::__construct` for the past 20 minutes: [fluorine:/a/mw-log] $ python /home/ori/cacheUpdateGrepper.py | dist Key|Ct (Pct) Histogram Wikibase\Repo\Api\ModifyEntity->attemptSaveEntity|6764 (27.07%) ██ SubmitAction->show|6311 (25.26%) █▋ ApiMain->executeActionWithErrorHandling|6291 (25.18%) █▌ Wikibase\Repo\Api\ModifyClaim->attemptSaveEntity| 940 (3.76%) █▎ ApiMain->execute| 803 (3.21%) ███▉ MediaWiki->performAction| 579 (2.32%) █▊ LoadBalancer->commitMasterChanges| 392 (1.57%) ███▉ EditAction->show| 331 (1.32%) ███▎ ApiMain->executeAction| 197 (0.79%) ██ FormlessAction->show| 150 (0.60%) █▌ Wikibase\Repo\Interactors\ItemMergeInteractor->mergeItems| 120 (0.48%) █▎ Wikibase\Repo\UpdateRepo\UpdateRepoJob->run| 106 (0.42%) █ TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Addshore, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, Denniss, aaron, faidon, Joe, ori, BBlack, Aklapper, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
ori added a comment. In https://phabricator.wikimedia.org/T124418#1963710, @JanZerebecki wrote: > https://grafana-admin.wikimedia.org/dashboard/db/tmp-t124418 What is that dashboard supposed to be showing? It is very hard to make sense of. TASK DETAIL https://phabricator.wikimedia.org/T124418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, Denniss, aaron, faidon, Joe, ori, BBlack, Aklapper, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori added a comment. https://phabricator.wikimedia.org/tag/wikidata/ folks (especially @daniel and @Lydia_Pintscher): Just a quick note to apologize for my conduct on this task. I think I was quick to escalate things, and I did it in a manner that probably only succeeded at raising the stress level of everyone involved. I'll treat it as a learning experience. :) TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo, ori Cc: Tobi_WMDE_SW, ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori added subscribers: mobrovac, ori. ori added a comment. @daniel, could you please make sure to sync up with the Services team to see how https://phabricator.wikimedia.org/T114443 could lay the groundwork for solving this problem better? @mobrovac, could you please act as liaison / point-person? TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T48643: [Story] Dispatching via delayed jobs (instead of cron script)
ori added a blocked task: T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections. TASK DETAIL https://phabricator.wikimedia.org/T48643 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, Tobi_WMDE_SW, JanZerebecki, Wikidata-bugs, Abraham, Nemo_bis, Denny, aude, Ricordisamoa, Lydia_Pintscher, daniel, hoo, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori added a comment. In https://phabricator.wikimedia.org/T118162#1795369, @jcrespo wrote: > - The cron jobs run as wikiadmin, which are on purpose not limited in > execution time, so there is no protection against them overflowing a database > server In https://phabricator.wikimedia.org/T118162#1795406, @daniel wrote: > - rewriting the dispatch logic to rely on the job queue has long been on the > todo list, see T48643: [Story] Dispatching via delayed jobs (instead of cron > script) <https://phabricator.wikimedia.org/T48643>. It's not trivial to get > right, though. Ok, so it sounds like we have: - A very severe problem. - Agreement on how to mitigate it. @Lydia_Pintscher, could you please make this a priority? Especially since the sprint goal of checking if this has been resolved has just been accomplished for you by Jaime. TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
ori added a blocking task: T48643: [Story] Dispatching via delayed jobs (instead of cron script). TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T116568: navigation timing stats specific to wikidata.org item? pages
ori removed a project: Performance-Team. TASK DETAIL https://phabricator.wikimedia.org/T116568 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JanZerebecki, ori Cc: Krinkle, gerritbot, thiemowmde, adrianheine, Addshore, JanZerebecki, Lydia_Pintscher, Aklapper, Ricordisamoa, Wikidata-bugs, aude ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T117398: number of database updates multiplied x3 since 29 November
ori added a subscriber: Performance-Team. ori set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T117398 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Performance-Team, Aklapper, jcrespo, Wikidata-bugs, aude, GWicke, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T114995: [Bug] Varnish cache invalidation not working for m.wikidata.org
ori closed this task as "Resolved". ori claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T114995 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: gerritbot, faidon, Addshore, BBlack, Jdlrobson, daniel, Lydia_Pintscher, Aklapper, hoo, Wikidata-bugs, aude, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP
ori added a comment. In https://phabricator.wikimedia.org/T114443#1701193, @GWicke wrote: > @Nuria, see the task description, heading "Initial use cases". Potential applications are one thing; a concise problem-statement another. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, mark, JanZerebecki, RobLa-WMF, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T113665: SitesModuleWorker::getSitesHash() is slow
ori edited projects, added Performance; removed Performance-Team. TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: adrianheine, Krenair, MaxSem, JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs, GWicke ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T113665: SitesModuleWorker::getSitesHash()
ori added a comment. SiteList (why can't it just be an array?) extends GenericArrayObject which extends ArrayObject. ArrayObject provides a `ksort()` method which would be useful here, except that SiteList maintains its own mappings of offsets to values in internal arrays. TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Krenair, MaxSem, JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T113665: SitesModuleWorker::getSitesHash()
ori created this task. ori added subscribers: ori, aude, hoo, daniel, JeroenDeDauw. ori added projects: Performance-Team, Wikidata. Herald added a subscriber: Aklapper. TASK DESCRIPTION Looking at performance data for `load.php` requests, I notice that we spend quite a lot of time in `SitesModuleWorker::getSitesHash()`. So I would like to know what that method does. I grep for it, and find that it calls `SitesModuleWorker::getSites()`. `SitesModuleWorker::getSites()`, in turn, calls `SiteStore::getSites()`. But SiteStore is an interface, and I don't know which concrete implementation SitesModuleWorker is using, because the implementation is specified via a parameter to `SitesModuleWorker::__construct()`. So I need to look and see where SitesModuleWorker is instantiated. That takes me to `lib/includes/modules/SitesModule.php:29`, where I see that it is `SiteSQLStore::newInstance()`. OK. But where is SiteSQLStore? Oh, I see that it's in Core. OK, let's pull it up and see what its `SiteSQLStore::getSites()` method does. Looks like it doesn't have one! SiteSQLStore extends CachingSiteStore, so I guess the method is implemented there. OK, let's go look at `CachingSiteStore::getSites()`. OK, so CachingSiteStore tries to get the sites from its cache object, which is a BagOStuff. I don't know which, because, again, that is up to the caller of `CachingSiteStore::__construct()`. Which is, it turns out, `SiteSQLStore::newInstance()`. But `SiteSQLStore::newInstance()` itself takes a `$cache` parameter. Fortunately, I remember that the caller is back in `SitesModule.php:29`, and I see that it does not specify a value for `$cache`, which means that it must be defaulting to `CACHE_ACCEL` for HHVM. Now I know what happens on a cache hit. Progress. But what about a cache miss? In that case, `CachingSiteStore::getSites()` calls `$this->siteStore->getSites();`. But what is its `$siteStore`? That is up to the caller of `CachingSiteStore::__construct()`. So I go back to `SiteSQLStore::newInstance()`, and find out that it is passing in a `DBSiteStore` instance. OK, let's go to DBSiteStore. `DBSiteStore::getSites()` calls `$this->loadSites();`. That leads me to `DBSiteStore::loadSites()`. Here, finally, I see an actual database query. All in all, to figure out what `SitesModuleWorker::getSitesHash()` is doing, I had to examine eight methods on no fewer than **six classes**. I still don't know why it is slow: cache hits and misses are not instrumented, and I haven't analyzed the query to see how the database executes it, but because I spent half an hour opening and closing files I am too exasperated to do that work. I have repeatedly suggested using a static array in `wmf-config` for this data, and at one point I believe this may have actually been implemented. Why are we not doing that? TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T113665: SitesModuleWorker::getSitesHash() is slow
ori changed the title from "SitesModuleWorker::getSitesHash()" to "SitesModuleWorker::getSitesHash() is slow". ori set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Krenair, MaxSem, JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T113665: SitesModuleWorker::getSitesHash()
ori added a subscriber: MaxSem. ori added a comment. @MaxSem looked at this code and instantly noticed that it is repeatedly sorting the cached value whenever it is retrieved (actually, whenever the method is called), and remarked that it would be far more sensible to do this once, when //saving// the value. TASK DETAIL https://phabricator.wikimedia.org/T113665 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: MaxSem, JeroenDeDauw, daniel, hoo, aude, ori, Aklapper, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Raised Priority] T111304: Fatal error: Call to undefined method OOUI\Element::serializeForApiResult() in /srv/mediawiki/php-1.26wmf21/vendor/oojs/oojs-ui/php/Element.php o
ori added subscribers: greg, ori. ori raised the priority of this task from "Normal" to "High". ori added a comment. This is *still* causing fatals in production. Can somebody fix this, please? @greg, could you please look after this? Anomie's suggestion from https://phabricator.wikimedia.org/T111304#1606372 is sound. TASK DETAIL https://phabricator.wikimedia.org/T111304 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: ori, greg, Jdlrobson, Anomie, Legoktm, Krenair, Jdforrester-WMF, Aklapper, mmodell, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Lydia_Pintscher, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Raised Priority] T58602: avoid fetching SiteList object from memcached
ori raised the priority of this task from High to Unbreak Now!. ori added a comment. Herald added a subscriber: Aklapper. Every so often I crunch some statistics about memcached usage. I can't tell you how demoralizing it is to find `sites/SiteList#2014-03-17+Site:2013-01-23` near the top again and again. If it's still there the next time I check I'm going to start disabling extensions. Changing priority to UBN! for that reason. Please fix this by making the list a static array in a PHP file in `operations/mediawiki-config` and then include it in `CommonSettings.php`. This way HHVM will compile it to byte-code and the OS will keep it in memory. TASK DETAIL https://phabricator.wikimedia.org/T58602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, daniel, Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride, csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, Unknown Object (MLST), GWicke, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T58602: avoid fetching SiteList object from memcached
ori added a comment. In https://phabricator.wikimedia.org/T58602#1463722, @hoo wrote: @aude is going to poke at this when she's back from Wikimania. Cool, thanks. TASK DETAIL https://phabricator.wikimedia.org/T58602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aude, ori Cc: Aklapper, daniel, Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride, csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, Unknown Object (MLST), GWicke, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T58602: avoid fetching SiteList object from memcached
ori added a comment. Thanks for the quick response. TASK DETAIL https://phabricator.wikimedia.org/T58602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aude, ori Cc: gerritbot, Aklapper, daniel, Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride, csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, Unknown Object (MLST), GWicke, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T88550: Mourn Titan
ori added a subscriber: ori. ori added a comment. In https://phabricator.wikimedia.org/T88550#1395307, @MZMcBride wrote: What replaced Titan? Blazegraph http://www.blazegraph.com/ TASK DETAIL https://phabricator.wikimedia.org/T88550 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: MaxSem, ori Cc: ori, MZMcBride, Krenair, Smalyshev, Legoktm, MaxSem, JanZerebecki, Aklapper, Manybubbles, jkroll, Wikidata-bugs, Jdouglas, aude ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Block] T76705: SiteStore / SiteList performance and caching (tracking)
ori reopened blocking task T58602: avoid fetching SiteList object from memcached as Open. TASK DETAIL https://phabricator.wikimedia.org/T76705 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, daniel, Wikidata-bugs, aude, GWicke, wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reopened] T58602: avoid fetching SiteList object from memcached
ori reopened this task as Open.
ori added a comment.
`{$wgDBname}:SiteList:sites/SiteList#2014-03-17+Site:2013-01-23` items are at
or near the top of memcached keys sorted by bandwidth utilization on the
production memcached cluster. This really needs to be fixed and stay fixed.
TASK DETAIL
https://phabricator.wikimedia.org/T58602
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Wikidata-bugs, ori
Cc: Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride,
csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, GWicke, wikidata-bugs
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Block] T76705: SiteStore / SiteList performance and caching (tracking)
ori reopened blocking task T58602: avoid fetching SiteList object from memcached as Open. TASK DETAIL https://phabricator.wikimedia.org/T76705 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, daniel, Wikidata-bugs, aude, GWicke, wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reopened] T58602: avoid fetching SiteList object from memcached
ori reopened this task as Open.
ori added a comment.
`{$wgDBname}:SiteList:sites/SiteList#2014-03-17+Site:2013-01-23` items are at
or near the top of memcached keys sorted by bandwidth utilization on the
production memcached cluster. This really needs to be fixed and stay fixed.
TASK DETAIL
https://phabricator.wikimedia.org/T58602
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Wikidata-bugs, ori
Cc: Wikidata-bugs, ori, JanZerebecki, aude, faidon, Ricordisamoa, MZMcBride,
csteipp, aaron, Lydia_Pintscher, jeremyb, hoo, GWicke, wikidata-bugs
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T86305: ApiStashEdit silently loses custom DataUpdates.
ori added a comment. @daniel Can you explain why this work needs to be done synchronously, blocking the user? TASK DETAIL https://phabricator.wikimedia.org/T86305 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ori Cc: Aklapper, daniel, aude, hoo, Lydia_Pintscher, aaron, ori, Jackmcbarn, Anomie, cscott, Wikidata-bugs, Jdforrester-WMF, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-tech] Article: Facebook's Top Open Data Problems
Facebook just published this summary of a summit for database researchers held at Menlo Park last September. I recommend it. It contains a clear and concise description of Facebook's data infrastructure, and a description of the open problems they are thinking about, which is even more interesting. https://research.facebook.com/blog/1522692927972019/facebook-s-top-open-data-problems/ To whet your appetite, here are the problems (the summaries mostly my own paraphrase): * Mobile: How should the shift toward mobile devices affect Facebook’s data infrastructure? * Reducing replication: How can we reduce the number of round trips between the application and data layers? * Impact of Caching on Availability (aka oh no, we just restarted memcached): How do we harness the efficiency gains provided by caching without being brought to our knees by a sudden drop in cache hit rate? * Sampling at logging time in a distributed environment: How should we sample log streams if we want to maintain accuracy and flexibility to answer post-hoc queries? * Trading storage space and CPU: TL;DR: gzip --best or gzip --fast? * Reliability of pipelines: Pipelines are less reliable than the sum of their parts. A pipeline composed of two systems, each 0.999 reliable, is 0.989 reliable. Much sadness. What to do? * Globally distributed warehouse: consistency models and synchronization problems. * Time series correlation and anomaly detection: AKA: I want an alert for that massive memcached bytes_out spike that doesn't also wake me up with false positives at 2AM. ___ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
Re: [Wikidata-tech] alternatives to memcached for caching entity objects across projects
On Wed, Oct 1, 2014 at 2:46 PM, Daniel Kinzler daniel.kinz...@wikimedia.de wrote: We currently use memcached to share cached objects across wikis, most importantly, entity objects (like data items). Ori suggested we should look into alternatives. This is what he wrote: [21:15] ori I was wondering if you think the way you use memcached is optimal (this sounds like a loaded question but I mean it sincerely). And if not, I was going to propose that you identify an optimal distributed object store, and I was also going to offer to help push for procurement and deployment of such a service on the WMF cluster. [21:17] ori memcached is a bit of a black box. it is very difficult to get comprehensible metrics about how much space and bandwidth you're utilizing, especially when your data is mixed up with everything else that goes into memcached [21:18] ori and the fact that you're serializing objects using php serialize() rather than simple values makes it even harder, because it means that you can only really poke around from php with wikidata code available The other major problem with memcache is that it doesn't support complex data structures like lists, queues, sets, or maps, so when you want to do things like, say, push or pop an item from a queue, you end up having to retrieve the entire collection, unserialize it, manipulate it locally, re-serialize it, and transmit it back in its entirety. Just food for thought, for now... any suggestions for a shared object store? I'm embarrassed to say that I don't know nearly enough about Wikidata to be able to make a recommendation. Where would you recommend I look if I wanted to understand the caching architecture? ___ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
[Wikidata-l] Memcached traffic
In November of last year, Aaron noticed a spike in memcached traffic, which we were eventually able to attribute to Wikibase client fetching the SiteList on every request. When Katie fixed it, outbound memcached traffic nearly halved. We're now back to the same rate of outbound traffic as then, and the culprit appears to be the same key. See: http://i.imgur.com/v9ebld6.png Tracked in https://bugzilla.wikimedia.org/show_bug.cgi?id=56602 ___ Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
[Wikidata-l] Fwd: [Engineering] Localisation not working on MediaWiki.org
See below for an extract of the discussion on the recurring disappearance of interface messages recently. It was a mistake for the discussion to unfold on an internal list, but it happened quite by chance, starting with an incident report and developing from there. --- Ori Livneh o...@wikimedia.org -- Forwarded message -- From: Ori Livneh o...@wikimedia.org Date: Thu, Apr 10, 2014 at 1:23 AM Subject: Re: [Engineering] Localisation not working on MediaWiki.org To: Brad Jorsch (Anomie) bjor...@wikimedia.org Cc: Bryan Davis bd...@wikimedia.org, Development and Operations Engineers engineer...@lists.wikimedia.org On Tue, Apr 8, 2014 at 6:56 AM, Brad Jorsch (Anomie) bjor...@wikimedia.orgwrote: On Mon, Apr 7, 2014 at 9:37 PM, Bryan Davis bd...@wikimedia.org wrote: The obvious change that caused this was that `mwversionsinuse --withdb` changed from returning 1.23wmf21=testwiki to 1.23wmf21=test2wiki. This result is used within scap by the mw-update-l10n script to run the maintenance script that builds the ExtensionMessages file. In theory the exact wiki passed to `mwscript mergeMessageFileList.php --wiki=WIKIDB` shouldn't matter, but obviously there are now some circumstances where it does indeed matter. It looks to me like it has always mattered to an extent: the final result from maintenance/mergeMessageFileList.php is the combination of extensions loaded for the --wiki wiki (e.g. in CommonSettings.php) and the extensions loaded by the script itself from the passed list of extensions. Hopefully the latter is always a superset of the former so that turns out not to matter. Interface messages went missing again on wikidata.org. l10nupdate ran updates on cawikibooks, where $wmgUseWikibaseClient is false. The theory that the exact wiki shouldn't make a difference is pretty shaky. You should expect to run on testwiki and fail loudly if you can't. We should rethink our whole approach; I don't have any confidence in the architecture. What is especially damning is not so much the recurrence of failures as the way they were discovered (that is to say: by chance) and the hard time we have had reasoning about their cause and the state of localization on the cluster generally. ___ Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
Re: [Wikidata-l] [Wikitech-l] Italian Wikipedia complements search results with Wikidata based functionality
On Mon, Dec 2, 2013 at 11:56 PM, Gerard Meijssen
gerard.meijs...@gmail.comwrote:
Hoi,
There are two ways of enabling this Wikidata search functionality; Wiki
wide and personally. It is done by adding this one line to common.js..
importScriptURI(//
en.wikipedia.org/w/index.php?title=MediaWiki:Wdsearch.jsaction=rawctype=text/javascript
);
You add it either to mediawiki:common.js or to USER/common.js.
Thanks,
GerardM
Please don't load the script on every page. It only targets the search
result page. This means a redundant request on every page that isn't the
search page. You can change it to:
if ( mw.config.get( 'wgCanonicalSpecialPageName' ) === 'Search' ) {
importScriptURI(//
en.wikipedia.org/w/index.php?title=MediaWiki:Wdsearch.jsaction=rawctype=text/javascript
);
}
Thanks!
Ori
___
Wikidata-l mailing list
Wikidata-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-l
