[Wikidata-bugs] [Maniphest] [Updated] T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007
hoo added a comment. So I finally managed to track this down: In Revision::getRevisionText, MainWANObjectCache::getWithSetCallback is used to retrieve revisions. As of 4432e898be078eea73fbf072f5a383ec1ded0d67 WANObjectCache uses a StatsdDataFactoryInterface for certain kinds of (often very high volume) logging. Given MediaWiki uses a BufferingStatsdDataFactory per default, we basically try to accumulate all these stat hits until we eventually run out of memory. Just commenting out $this->buffer[] = $entity; in BufferingStatsdDataFactory::produceStatsdData (locally on snapshot1001) immediately fixed this.TASK DETAILhttps://phabricator.wikimedia.org/T181385EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: TerraCodes, Jay8g, Liuxinyu970226, Stashbot, Aklapper, daniel, Lydia_Pintscher, ArielGlenn, aude, hoo, Lahi, Gq86, GoranSMilovanovic, QZanden, Wikidata-bugs, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007
hoo edited projects, added Performance-Team; removed MediaWiki-extensions-WikibaseRepository. TASK DETAILhttps://phabricator.wikimedia.org/T181385EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaron, hooCc: Legoktm, Addshore, Krinkle, demon, TerraCodes, Jay8g, Liuxinyu970226, Stashbot, Aklapper, daniel, Lydia_Pintscher, ArielGlenn, aude, hoo, Imarlier, Lahi, Gq86, GoranSMilovanovic, QZanden, Vali.matei, Wikidata-bugs, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007
gerritbot added a project: Patch-For-Review. TASK DETAILhttps://phabricator.wikimedia.org/T181385EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaron, gerritbotCc: gerritbot, Legoktm, Addshore, Krinkle, demon, TerraCodes, Jay8g, Liuxinyu970226, Stashbot, Aklapper, daniel, Lydia_Pintscher, ArielGlenn, aude, hoo, Imarlier, Lahi, Gq86, Baloch007, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Maathavan, Wikidata-bugs, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007
ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2017-12-05 (1.31.0-wmf.11)). TASK DETAILhttps://phabricator.wikimedia.org/T181385EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaron, ReleaseTaggerBotCc: Envlh, gerritbot, Legoktm, Addshore, Krinkle, demon, TerraCodes, Jay8g, Liuxinyu970226, Stashbot, Aklapper, daniel, Lydia_Pintscher, ArielGlenn, aude, hoo, Imarlier, Lahi, Gq86, Baloch007, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Maathavan, Wikidata-bugs, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007
ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)); removed MW-1.31-release-notes (WMF-deploy-2017-12-05 (1.31.0-wmf.11)). TASK DETAILhttps://phabricator.wikimedia.org/T181385EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaron, ReleaseTaggerBotCc: Envlh, gerritbot, Legoktm, Addshore, Krinkle, demon, TerraCodes, Jay8g, Liuxinyu970226, Stashbot, Aklapper, daniel, Lydia_Pintscher, ArielGlenn, aude, hoo, Imarlier, Lahi, Gq86, Baloch007, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Maathavan, Wikidata-bugs, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007
Smalyshev added a comment. We had similar problems with long-running scripts and statsd here: T181716: forceSearchIndex.php hangs at the end of the process when running on large wikis. I think combinantion of statsd and long-running maintenance scripts is not working well with some recent change there. Hopefully limiting it to non-CLI will fix it.TASK DETAILhttps://phabricator.wikimedia.org/T181385EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaron, SmalyshevCc: Smalyshev, Envlh, gerritbot, Legoktm, Addshore, Krinkle, demon, TerraCodes, Jay8g, Liuxinyu970226, Stashbot, Aklapper, daniel, Lydia_Pintscher, ArielGlenn, aude, hoo, Imarlier, Lahi, Gq86, Baloch007, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Maathavan, Wikidata-bugs, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007
ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)); removed MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)). TASK DETAILhttps://phabricator.wikimedia.org/T181385EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, ReleaseTaggerBotCc: aaron, Smalyshev, Envlh, gerritbot, Legoktm, Addshore, Krinkle, demon, TerraCodes, Jay8g, Stashbot, Aklapper, daniel, Lydia_Pintscher, ArielGlenn, aude, hoo, Imarlier, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, Wikidata-bugs, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
