[Wikidata-bugs] [Maniphest] [Updated] T187521: Optimize recentchanges and wbc_entity_usage table across wikis
jcrespo added a subtask: T178290: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki. TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, GoranSMilovanovic, Marostegui, Aklapper, revi, Lydia_Pintscher, Ladsgroup, Lahi, Gq86, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T187521: Optimize recentchanges and wbc_entity_usage table across wikis
jcrespo added a subtask: T12: Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata). TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, GoranSMilovanovic, Marostegui, Aklapper, revi, Lydia_Pintscher, Ladsgroup, Lahi, Gq86, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T177772: Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata)
jcrespo added a parent task: T187521: Optimize recentchanges and wbc_entity_usage table across wikis. TASK DETAILhttps://phabricator.wikimedia.org/T12EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: gerritbot, Stashbot, Jdforrester-WMF, Bawolff, Lydia_Pintscher, hoo, Ladsgroup, Base, Aklapper, TerraCodes, Jay8g, Marostegui, jcrespo, Majesticalreaper22, Ahmed123, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, GoranSMilovanovic, Adik2382, Abiyoyo, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Jack_who_built_the_house, Lewizho99, Minhnv-2809, Maathavan, Iniquity, Luke081515, Wikidata-bugs, aude, putnik, Mbch331, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T187521: Optimize recentchanges and wbc_entity_usage table across wikis
jcrespo added a comment. probably commonswiki and ruwiki were done at T12. The list of the ones scheduled to do next was at T178290#3688203TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, GoranSMilovanovic, Marostegui, Aklapper, revi, Lydia_Pintscher, Ladsgroup, Lahi, Gq86, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T190153: In the slots table, replace slot_inherited with slot_origin
jcrespo added a comment. Whoever deployed the table, violated the advice when creating new tables: "However, due to labs filtering limitation (hopefully, to be solved soon) you still should block on a DBA to review the table creation to check data leak is not possible." https://wikitech.wikimedia.org/wiki/Schema_changes I would like to know who and how that happened?TASK DETAILhttps://phabricator.wikimedia.org/T190153EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, aude, Addshore, Aklapper, Anomie, Tgr, daniel, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, Minhnv-2809, Agabi10, Susannaanas, Luke081515, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T190153: In the slots table, replace slot_inherited with slot_origin
jcrespo added a comment. Ok, Manuel haven't communicated to me and I wasn't on that ticket. Apparently the tables were created too soon.TASK DETAILhttps://phabricator.wikimedia.org/T190153EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, aude, Addshore, Aklapper, Anomie, Tgr, daniel, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, Minhnv-2809, Agabi10, Susannaanas, Luke081515, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T190153: In the slots table, replace slot_inherited with slot_origin
jcrespo added a comment. +1 in case there are more changesTASK DETAILhttps://phabricator.wikimedia.org/T190153EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, aude, Addshore, Aklapper, Anomie, Tgr, daniel, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, Minhnv-2809, Agabi10, Susannaanas, Luke081515, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T190153: In the slots table, replace slot_inherited with slot_origin
jcrespo added a comment. "too soon" in that the structure was not stable, even if it was merged.TASK DETAILhttps://phabricator.wikimedia.org/T190153EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, aude, Addshore, Aklapper, Anomie, Tgr, daniel, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, Minhnv-2809, Agabi10, Susannaanas, Luke081515, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T190667: Table langlinks shows outdated links
jcrespo removed a project: Data-Services.jcrespo added a comment. @Yamaha5 I think I know what is the issue- A wikidata edit was done, but the page still have interwiki links, see at the end of: https://en.wikipedia.org/w/index.php?title=Category:Australian_suffragists&oldid=624098262&action=""> I wonder if those being there are having issues with wikidata on parsing.TASK DETAILhttps://phabricator.wikimedia.org/T190667EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Chicocvenancio, Yamaha5, jcrespo, zhuyifei1999, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Giuliamocci, 1978Gage2001, aborrero, Tbscho, JJMC89, srodlund, Gryllida, scfc, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T188279: Investigate optimzing wb_terms
jcrespo added subscribers: Marostegui, jcrespo.jcrespo added a comment. @Ladsgroup @Lucas_Werkmeister_WMDE @Marostegui Let's talk here about provinding a temporary wikidata replica so wikidata team can play with changes to it.TASK DETAILhttps://phabricator.wikimedia.org/T188279EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, Nikerabbit, Liuxinyu970226, Edgars2007, thiemowmde, Lucas_Werkmeister_WMDE, Aklapper, WMDE-leszek, RazShuty, daniel, Lydia_Pintscher, Ladsgroup, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T188279: Investigate optimzing wb_terms
jcrespo added a project: DBA.jcrespo added a comment. Adding #DBA to support the setup of this test db.TASK DETAILhttps://phabricator.wikimedia.org/T188279EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, Nikerabbit, Liuxinyu970226, Edgars2007, thiemowmde, Lucas_Werkmeister_WMDE, Aklapper, WMDE-leszek, RazShuty, daniel, Lydia_Pintscher, Ladsgroup, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188279: Investigate optimzing wb_terms
jcrespo added a comment. We can either change the password, or the user, or both. +1TASK DETAILhttps://phabricator.wikimedia.org/T188279EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Stashbot, gerritbot, jcrespo, Marostegui, Nikerabbit, Liuxinyu970226, Edgars2007, thiemowmde, Lucas_Werkmeister_WMDE, Aklapper, WMDE-leszek, RazShuty, daniel, Lydia_Pintscher, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T191626: Remove term_entity_type from wb_terms
jcrespo removed a project: DBA.jcrespo added a comment. Wikidata team will test it on the test host and create a task for production deployment with all requested changes/full strategy. Until then, there is nothing for us to do here. If you need help, re-add us with a specific request.TASK DETAILhttps://phabricator.wikimedia.org/T191626EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Ladsgroup, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Marostegui, Minhnv-2809, Luke081515, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T191391: Apply schema changes to an isolated database and examine the results
jcrespo added a comment. I found T86530, which may be outdated, but may help with giving more options. As a note to @Ladsgroup, and the other people working on this- after dropping a column, replication stopped working on the test host (as expected). This is not a problem, just a remark so you know why no more updates are arriving to that host (and of course, that in production, the change should be done in a different compatible way first). It would be nice to report size changes here, so we are all aware.TASK DETAILhttps://phabricator.wikimedia.org/T191391EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, Marostegui, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T187521: Optimize recentchanges and wbc_entity_usage table across wikis
jcrespo added a comment. commonswiki errors due to deadlocks on INSERT IGNORE INTO wbc_entity_usage seem to be common (not too worrying, but on of the most comon database errors), could the code be optimized to avoid those? I am guessing that the same row is written many times (once per change on the same item), and maybe that could be simplified somehow. INSERT IGNORE is a bit of a bad trick here, and we may be writing multiple times the same data without need. Given the changes are done by the job queue and arrive in any order, maybe transaction serialization can be relaxed?TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, GoranSMilovanovic, Marostegui, Aklapper, revi, Lydia_Pintscher, Ladsgroup, Lahi, Gq86, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T192349: deadlocks on INSERT IGNORE INTO wbc_entity_usage
jcrespo created this task.jcrespo added projects: Wikidata, Wikimedia-log-errors.Herald added a subscriber: Aklapper. TASK DESCRIPTIONSeparating from T187521#4133553 commonswiki errors due to deadlocks on INSERT IGNORE INTO wbc_entity_usage seem to be common (not too worrying, but on of the most comon database errors), could the code be optimized to avoid those? I am guessing that the same row is written many times (once per change on the same item), and maybe that could be simplified somehow. INSERT IGNORE is a bit of a bad trick here, and we may be writing multiple times the same data without need. Given the changes are done by the job queue and arrive in any order, maybe transaction serialization can be relaxed? https://logstash.wikimedia.org/goto/20defb50b256c8da0ddbf3a3bb258017TASK DETAILhttps://phabricator.wikimedia.org/T192349EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Ladsgroup, Aklapper, jcrespo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T187521: Optimize recentchanges and wbc_entity_usage table across wikis
jcrespo added a comment. Done as T192349TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, GoranSMilovanovic, Marostegui, Aklapper, revi, Lydia_Pintscher, Ladsgroup, Lahi, Gq86, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium
jcrespo added a comment. Depooled db1109 2 days ago, I still cannot put it under maintenance. I still see it on other sections other than s8, see T143870. If the code doesn't work, special mediawiki configuration should be setup so that dump hosts only know about dump dbs, I can see that working fine. I would like the ability to tell LB to drop the current connection AND config and re-read. This would be very handy in general. That functionality was added when we had issues with commons transcoding leaving db connections open, AFAIKTASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, gnosygnu, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T189596: Run deleteAutopatrolLogs script for Wikidata (WMF)
jcrespo added a comment. @Ladsgroup after the table rebuilding on dbstore2001, purging is going at a good pace: https://grafana.wikimedia.org/dashboard/db/mysql?panelId=11&fullscreen&orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=dbstore2001&var-port=13318&from=1524683251842&to=1524726451842 As soon as the counter gets close to 0, the row deletion can continue. At the current pace I predict it will be around 1 day of waitinf, if purge keeps happening at the same rate.TASK DETAILhttps://phabricator.wikimedia.org/T189596EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: jcrespo, GeoffreyT2000, kaldari, gerritbot, Stashbot, Lucas_Werkmeister_WMDE, Aklapper, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Dcljr, Lydia_Pintscher, Mbch331, Rxy, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T191282: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending
jcrespo added a project: Wikidata.jcrespo added a comment. Most of the warnings (I suppose the logging was changed to understand which component relates to) seem to be related to wikibase client jobs right now: https://logstash.wikimedia.org/app/kibana#/dashboard/DBQueryTASK DETAILhttps://phabricator.wikimedia.org/T191282EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: gerritbot, Tgr, Pchelolo, mmodell, Marostegui, aaron, mobrovac, Aklapper, jcrespo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Eevans, JAllemandou, Hardikj, Wikidata-bugs, aude, GWicke, Mbch331, Jay8g, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T174047: Provide backwards compatibility views for toolforge replica [MCR]
jcrespo added a comment. Full table scans are indeed expected and preferred when tables have 0 to a only a few rows. However, I think Bstorm is right to be concerned, we have seen lately an increase in load on labsdbs, but I have yet to check if it is related to the usage of views or just coincidental.TASK DETAILhttps://phabricator.wikimedia.org/T174047EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, gerritbot, Bstorm, Tgr, bd808, Anomie, Aklapper, daniel, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tbscho, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, srodlund, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T191391: Apply schema changes to an isolated database and examine the results
jcrespo added a comment. The indexes seem disproportionally large compared to the data. Could the table be split somehow in a 1:1 relationship that could make sense (e.g. index_table with just columns to search and data_table with almost just the primary index).TASK DETAILhttps://phabricator.wikimedia.org/T191391EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, Marostegui, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T191391: Apply schema changes to an isolated database and examine the results
jcrespo added a comment. Yes, that is much better.TASK DETAILhttps://phabricator.wikimedia.org/T191391EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, Marostegui, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194270: Drop 'tmp1' index from wb_terms table in production
jcrespo added a comment. This would have been very useful in this case https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.htmlTASK DETAILhttps://phabricator.wikimedia.org/T194270EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Marostegui, RazShuty, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T191282: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending
jcrespo added a comment. Most if not all of these seem to have gone since yesterday's train (need more time to check if absolutly all are gone or just the bulk of it).TASK DETAILhttps://phabricator.wikimedia.org/T191282EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Joe, Ladsgroup, hoo, RazShuty, gerritbot, Tgr, Pchelolo, mmodell, Marostegui, aaron, mobrovac, Aklapper, jcrespo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Eevans, JAllemandou, Hardikj, Wikidata-bugs, aude, GWicke, Mbch331, Jay8g, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Lowered Priority] T191282: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending
jcrespo lowered the priority of this task from "High" to "Normal".jcrespo added a comment. @mmodell Most of the errors are gone, but some are still happening. I think this is no longer high, but if it has an easy fix, those should be looked at by the code owners: https://logstash.wikimedia.org/goto/dd624348a6d23ce0776f4cc66a0eac7eTASK DETAILhttps://phabricator.wikimedia.org/T191282EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Joe, Ladsgroup, hoo, RazShuty, gerritbot, Tgr, Pchelolo, mmodell, Marostegui, aaron, mobrovac, Aklapper, jcrespo, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Eevans, JAllemandou, Hardikj, Wikidata-bugs, aude, GWicke, Mbch331, Jay8g, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198049: Investigate possible outage on wikidata on 25th June - 04:13AM UTC - 05:27AM UTC
jcrespo added a comment. I am not too worried about exceptions/error messages, I only pointed those in case it helped debug the real issues, the ones I mentioned at T198049#4310282 (in first appearance caused by a single production host having issues).TASK DETAILhttps://phabricator.wikimedia.org/T198049EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: tstarling, jcrespo, hoo, Lydia_Pintscher, daniel, Ladsgroup, Marostegui, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. Let's keep an eye on performance metrics, specially https://grafana.wikimedia.org/dashboard/db/save-timing Although with the estimations given T183488#4480579, it may be desirable to make things faster rather and short than extended for a long period of time. Meanwhile, I will keep an eye on the disk usage and size.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reopened] T147169: Make sure Wikibase dump maintenance scripts solely use the "dump" db group
jcrespo reopened this task as "Open".jcrespo added a comment. dumpJson.php is failing many times per minute, while I only rebooted for maintenance an rc replica (which is fully depooled): https://logstash.wikimedia.org/goto/e2dee89b02bc72774c4320671d879b05TASK DETAILhttps://phabricator.wikimedia.org/T147169EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Lucas_Werkmeister_WMDE, ArielGlenn, thiemowmde, Addshore, aaron, Aklapper, Smalyshev, Lydia_Pintscher, jcrespo, aude, daniel, hoo, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, LawExplorer, Salgo60, Jonas, Wikidata-bugs, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Block] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium
jcrespo reopened subtask T147169: Make sure Wikibase dump maintenance scripts solely use the "dump" db group as "Open". TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, gnosygnu, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T147169: Make sure Wikibase dump maintenance scripts solely use the "dump" db group
jcrespo added a comment. does not try to refresh it periodically But db1101 never was a dump host, but rc, and it used to have load 1.TASK DETAILhttps://phabricator.wikimedia.org/T147169EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Lucas_Werkmeister_WMDE, ArielGlenn, thiemowmde, Addshore, aaron, Aklapper, Smalyshev, Lydia_Pintscher, jcrespo, aude, daniel, hoo, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, LawExplorer, Salgo60, Jonas, Wikidata-bugs, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T147169: Make sure Wikibase dump maintenance scripts solely use the "dump" db group
jcrespo added a comment. No need to be sorry, could this be a load balancer bug?TASK DETAILhttps://phabricator.wikimedia.org/T147169EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Lucas_Werkmeister_WMDE, ArielGlenn, thiemowmde, Addshore, aaron, Aklapper, Smalyshev, Lydia_Pintscher, jcrespo, aude, daniel, hoo, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, LawExplorer, Salgo60, Jonas, Wikidata-bugs, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. @tstarling Please stop writes going to *s2* unless they have already finished, there are plans to put s2 in read only without much pre-warning (unscheduled read only) T201694- I don't know exactly what is going to be the plan (a logical failover or a network maintenance), but probably we should preventively pause all long-running write traffic to s2 until I know the details and it is safe to continue writing in batches.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T147169: Make sure Wikibase dump maintenance scripts solely use the "dump" db group
jcrespo added a comment. I still have no idea how this could happen (also these errors don't seem to be on mwlog1001?!). Does dumpJson.php call any function that could force any check on all replicas to fail (and because it has old configuration, it fails on a depooled non-slow host?TASK DETAILhttps://phabricator.wikimedia.org/T147169EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Lucas_Werkmeister_WMDE, ArielGlenn, Addshore, aaron, Aklapper, Smalyshev, Lydia_Pintscher, jcrespo, aude, daniel, hoo, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, LawExplorer, Salgo60, Jonas, Wikidata-bugs, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T202032: Duplicate ar_rev_id values in several wikis
jcrespo added a comment. Related: T135851 T164488#3530600 T162593#3175313 T162807#3882309 T163190#3246365 T160509#3110389TASK DETAILhttps://phabricator.wikimedia.org/T202032EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: Aklapper, daniel, aude, Addshore, Anomie, Abit, jcrespo, tstarling, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, JJMC89, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. I mentioned this to Tim, and he said to better mention it to Brad (and Daniel?), that it would be nice to run a read-only process to check the consistency of the migration- it should take much less time than the writes and would avoid headaches given the issues with the existing data and other possible complications that could have happened during the migration. I don't think anything is problem-free, but we can try to detect the issues as much as we can. What do you think?TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. In T183488#4506824, @daniel wrote: We did pretty extensive testing of this on the test copies (db and db1112). We could have a script that does basic sanity checks, but I'm not sure what to test, beyond the trivial. All I can think of is: all revision rows have an associated slot row. all archive rows have an associated slot row. all slot rows have an associated content. ar_rev_id as no conflicts with rev_id ar_rev_id as no conflicts with itself Is that what you had in mind? Yes, also the inverse checks, assuming it is 1:1 at the moment.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. So the script will make sure to not miss any content, but doesn't assure it introduces extra one, or duplicates some, that is why I mention the usage of extra checks. Takes very little time (1 long running query per check) and validates the whole process. The same thing that "all code should have tests" "all tasks should have a way to validate they completed correctly". I think this is very process to miss that, don't you think?TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T202483: www.mediawiki.org showing: error:Unknown database 'wikidatawiki' on shard: s3
jcrespo updated the task description. (Show Details)jcrespo added a project: Wikidata.
CHANGES TO TASK DESCRIPTIONWe are seeing lots of errors related with the following output (https://logstash.wikimedia.org/goto/f8ec4b3e22a8b3d5a8ad0143a4f8982f):
```
server:www.mediawiki.org unique_id:W3z0zApAICkAAFWf3DoH db_server:10.64.48.35 method:Wikimedia\Rdbms\DatabaseMysqlBase::open level:ERROR ip:10.64.48.103 wiki:mediawikiwiki channel:DBConnection mwversion:1.32.0-wmf.18 message:Error connecting to 10.64.48.35: Unknown database 'wikidatawiki' type:mediawiki error:Unknown database 'wikidatawiki' normalized_message:Error connecting to {db_server}: {error} url:/wiki/Special:MyLanguage/Help:Logging_in tags:syslog, es, es reqId:W3z0zApAICkAAFWf3DoH referrer:http://wiki.wcpl.info/w/index.php?title=Special:UserLogin&returnto=Main+Page @timestamp:2018-08-22T05:29:48 http_method:GET db_name:wikidatawiki db_user:wikiuser @version:1 host:mw1320 shard:s3 _id:AWVgHD8JoOODFPKvaCfH _type:mediawiki _index:logstash-2018.08.22 _score: -```lines=10
{
"_index": "logstash-2018.08.22",
"_type": "mediawiki",
"_id": "AWVgIepq8VFZOIjHYi4M",
"_version": 1,
"_score": null,
"_source": {
"server": "www.mediawiki.org",
"unique_id": "W3z2PwpAAEUAADLO8AUR",
"db_server": "10.64.32.136",
"method": "Wikimedia\\Rdbms\\DatabaseMysqlBase::open",
"level": "ERROR",
"ip": "10.64.48.103",
"wiki": "mediawikiwiki",
"channel": "DBConnection",
"mwversion": "1.32.0-wmf.18",
"message": "Error connecting to 10.64.32.136: Unknown database 'wikidatawiki'",
"type": "mediawiki",
"error": "Unknown database 'wikidatawiki'",
"normalized_message": "Error connecting to {db_server}: {error}",
"url": "/wiki/Special:MyLanguage/Help:Logging_in",
"tags": [
"syslog",
"es",
"es"
],
"reqId": "W3z2PwpAAEUAADLO8AUR",
"referrer": "http://hydrodictyon.eeb.uconn.edu/eebedia/index.php?title=Special:UserLogin&returnto=Main+Page",
"@timestamp": "2018-08-22T05:36:00.000Z",
"http_method": "GET",
"db_name": "wikidatawiki",
"db_user": "wikiuser",
"@version": 1,
"host": "mw1274",
"shard": "s3"
},
"fields": {
"@timestamp": [
153491616
]
},
"sort": [
153491616
]
}
```...TASK DETAILhttps://phabricator.wikimedia.org/T202483EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Paladox, Aklapper, greg, Liuxinyu970226, Prtksxna, TerraCodes, Marostegui, Lahi, Gq86, GoranSMilovanovic, RazeSoldier, QZanden, LawExplorer, Liudvikas, Luke081515, Wikidata-bugs, aude, zeljkofilipin, KartikMistry, Mbch331, Rxy, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T202483: www.mediawiki.org showing: error:Unknown database 'wikidatawiki' on shard: s3
jcrespo added a comment. beta has a single shard s/shard/section/TASK DETAILhttps://phabricator.wikimedia.org/T202483EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Stashbot, gerritbot, Addshore, Anomie, daniel, tstarling, Paladox, Aklapper, greg, Liuxinyu970226, Prtksxna, TerraCodes, Marostegui, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, RazeSoldier, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Liudvikas, Luke081515, Wikidata-bugs, aude, zeljkofilipin, KartikMistry, Mbch331, Rxy, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T189101: Deploy schema change for adding numeric primary key to wbqc_constraints table
jcrespo added a comment. @Lucas_Werkmeister_WMDE What is the plan with constraint_guid? Will you drop it? Shouldn't it keep the UNIQUE constraint while it is in use (and by extension, the index)? We need a quick answer if this is to be done in September.TASK DETAILhttps://phabricator.wikimedia.org/T189101EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, Ladsgroup, Aklapper, Jonas, hoo, aude, gerritbot, Lucas_Werkmeister_WMDE, Lahi, Gq86, GoranSMilovanovic, QZanden, merbst, LawExplorer, Agabi10, Luke081515, Wikidata-bugs, Mbch331, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T189101: Deploy schema change for adding numeric primary key to wbqc_constraints table
jcrespo added a comment. Please if you could give this a lot of priority, because if we miss this "train" (switchover) we could be waiting for a full extra year. Thanks. PK changes are not easy to do normally.TASK DETAILhttps://phabricator.wikimedia.org/T189101EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, Ladsgroup, Aklapper, Jonas, hoo, aude, gerritbot, Lucas_Werkmeister_WMDE, Lahi, Gq86, GoranSMilovanovic, QZanden, merbst, LawExplorer, Agabi10, Luke081515, Wikidata-bugs, Mbch331, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T202764: Wikidata produces a lot of failed requests for recentchanges API
jcrespo added a comment.
I got this on mediawiki logs at the same time than one of the retries. Are you sure you are not using a deprecated query? I don't know if they are correlated, but it happened in the same time frame and it is json related.
Warning: API call had warnings trying to get remote JsonConfig: warnings={"main":{"*":"Subscribe to the mediawiki-api-announce mailing list at \u003Chttps://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce\u003E for notice of API deprecations and breaking changes. Use [[Special:ApiFeatureUsage]] to see usage of deprecated features by your application."},"revisions":{"*":"Because \"rvslots\" was not specified, a legacy format has been used for the output. This format is deprecated, and in the future the new format will always be used."}}, query={"action":"query","titles":"Data:F3 \u0026 F4 shared wharves.map","prop":"revisions","rvprop":"content","continue":""} [Called from JsonConfig\JCUtils::warn in /srv/mediawiki/php-1.32.0-wmf.18/extensions/JsonConfig/includes/JCUtils.php at line 53] in /srv/mediawiki/php-1.32.0-wmf.18/includes/debug/MWDebug.php on line 309
However, that should be only a warning, the other, more probable cause is:
Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/client/includes/Usage/UsageAspectTransformer.php on line 175
Which would fit better, it is wikidatawiki specifically, and would fit better with things failing just sometimes.TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Imarlier, Ladsgroup, Lydia_Pintscher, WMDE-leszek, Aklapper, Gehel, Smalyshev, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Vali.matei, Zppix, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T202764: Wikidata produces a lot of failed requests for recentchanges API
jcrespo added a comment. @Ladsgroup That could be a separate issue and could be handled on a separate task (I don't know). The on topic here is UsageAspectTransformer.php (probably you saw that, just in case my comment was misleading about what was the main actionable here).TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Yurik, jcrespo, Imarlier, Ladsgroup, Lydia_Pintscher, WMDE-leszek, Aklapper, Gehel, Smalyshev, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Vali.matei, Zppix, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. Thank you anomie for running those queries, which I had suggested or even offered to do to check things look consistent.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T174047: Hide deprecated/unused fields on toolforge replica [MCR]
jcrespo added a comment. Yeah, let's announce it first as this time is a "breaking change" and we can wait a bit to deploy. I have also pending to to a rolling restart for mariadb upgrade and we can do it at the same time as otherwise for large tables all changes may fail because of metadata locking. BTW, if you think it is ok, we can also amend wikireplica documentation indicating that we try, but cannot guarantee backwards compatibility on db schema, and if people don't want to maintain the changes frequently, the API is the most stable method.TASK DETAILhttps://phabricator.wikimedia.org/T174047EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Stashbot, jcrespo, gerritbot, Bstorm, Tgr, bd808, Anomie, Aklapper, daniel, AndyTan, Gaboe420, Zylc, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, 1978Gage2001, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tbscho, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, srodlund, Wikidata-bugs, aude, Gryllida, scfc, Jdforrester-WMF, Mbch331, Ltrlg, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T174047: Hide deprecated/unused fields on toolforge replica [MCR]
jcrespo added a comment. will anything break I don't think because fields have been added but not yet removed, but I cannot promise that will happen soon or it is happening now (we should test on deploy on a smaller db first).TASK DETAILhttps://phabricator.wikimedia.org/T174047EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Stashbot, jcrespo, gerritbot, Bstorm, Tgr, bd808, Anomie, Aklapper, daniel, AndyTan, Gaboe420, Zylc, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, 1978Gage2001, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tbscho, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, srodlund, Wikidata-bugs, aude, Gryllida, scfc, Jdforrester-WMF, Mbch331, Ltrlg, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T174047: Hide deprecated/unused fields on toolforge replica [MCR]
jcrespo added a comment. I've written this: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Stability_of_the_mediawiki_database_schema please review if it makes sense and ping if you see issues or edit it yourself and it would be great if you could send an email referencing this to cloud-l. If it isn't breaking The views shouldn't be breaking, but I am not so sure about the tools using them... the reboot depools. That is not likely happening this week because of the dc switchover :-(TASK DETAILhttps://phabricator.wikimedia.org/T174047EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Stashbot, jcrespo, gerritbot, Bstorm, Tgr, bd808, Anomie, Aklapper, daniel, AndyTan, Gaboe420, Zylc, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, 1978Gage2001, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tbscho, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, srodlund, Wikidata-bugs, aude, Gryllida, scfc, Jdforrester-WMF, Mbch331, Ltrlg, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T174047: Hide deprecated/unused fields on toolforge replica [MCR]
jcrespo added a comment. @daniel look at the writeup I mention above, and see if it seems reasonable- note that tools may rely on old structure, in terms of fields may not be dropped yet, but I guess they will eventually be emptied. Note that the main issues we learned is that compatibility views created issues due to unoptimized queries- and there was no way to access the real underlying structure.TASK DETAILhttps://phabricator.wikimedia.org/T174047EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Stashbot, jcrespo, gerritbot, Bstorm, Tgr, bd808, Anomie, Aklapper, daniel, AndyTan, Gaboe420, Zylc, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, 1978Gage2001, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tbscho, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, srodlund, Wikidata-bugs, aude, Gryllida, scfc, Jdforrester-WMF, Mbch331, Ltrlg, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T174047: Hide deprecated/unused fields on toolforge replica [MCR]
jcrespo added a comment. no issue with scwiki or any other small one- you may still need he depooling to avoid headaches on the larger ones :-)TASK DETAILhttps://phabricator.wikimedia.org/T174047EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Stashbot, jcrespo, gerritbot, Bstorm, Tgr, bd808, Anomie, Aklapper, daniel, AndyTan, Gaboe420, Zylc, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, 1978Gage2001, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tbscho, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, srodlund, Wikidata-bugs, aude, Gryllida, scfc, Jdforrester-WMF, Mbch331, Ltrlg, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T204531: Wikidata dumps creating large amounts of log spam
jcrespo created this task.jcrespo added projects: Wikimedia-production-error, Dumps-Generation, Wikidata, MediaWiki-Logging. TASK DESCRIPTION(Not sure yet if these are the regular dumps or the wikidata-specific ones) Lots of errors coming from snapshot1008 related to replication lag and read only mode. This is probably caused because the eqiad s8 master (passive datacenter) was being reimaged for upgrade to the latest kernel and security patches and the mariadb 10.1 upgrade. (Jaime's comment: This happens once every 3 years.) We need to research why this was happening, and preventing on some way: Can server maintenance be done on a better way, not creating log spam? Can replication checks reduce its spam-icity so we get properly alerted but don't get the same message millions of time. Eg, give an error, but not on cached subsequent occurences or other different model Why does the snapshots require not being lagged? Is lag important for those? If no, Can they just skip those checks? If yes, shouldn't they be running on the active datacenter instead, and not on eqiad? While this didn't cause issues to final issues, this cause some issues on production due to impact to the logging infrastructure: https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1TASK DETAILhttps://phabricator.wikimedia.org/T204531EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, Krinkle, fgiunchedi, jcrespo, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, MGChecker, gnosygnu, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T204531: Wikidata dumps creating large amounts of log spam
jcrespo updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION...While this didn't cause issues to final issues, this cause some issues on production due to impact to the logging infrastructure: https://grafana.wikimedia.org/dashboard/db/production-logging?orgId=1&from=1537184523025&to=1537197061054 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&from=1537184523025&to=1537197061054TASK DETAILhttps://phabricator.wikimedia.org/T204531EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Agabi10, Marostegui, ArielGlenn, Krinkle, fgiunchedi, jcrespo, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, MGChecker, gnosygnu, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T202764: Wikidata produces a lot of failed requests for recentchanges API
jcrespo added a project: Datacenter-Switchover-2018. TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Marostegui, Banyek, Reedy, gerritbot, Krinkle, Addshore, Yurik, jcrespo, Imarlier, Ladsgroup, Lydia_Pintscher, WMDE-leszek, Aklapper, Gehel, Smalyshev, jijiki, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Volans, Maathavan, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, Krenair, akosiaris, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T202764: Wikidata produces a lot of failed requests for recentchanges API
jcrespo added a comment. The plan is for us dbas to test setting up a single API with the same structure than eqiad and do all assuming that fixies it, and later we will have to evaluate what is the right long-term status, given some unknowns and related tasks such as T202167: One note: This index might be already existing but under name of tmp_1 or tmp_2 or tmp_3. I don't know how complex renaming an index is but that'd be useful.TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Marostegui, Banyek, Reedy, gerritbot, Krinkle, Addshore, Yurik, jcrespo, Imarlier, Ladsgroup, Lydia_Pintscher, WMDE-leszek, Aklapper, Gehel, Smalyshev, jijiki, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Volans, Maathavan, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, Krenair, akosiaris, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T204531: Wikidata dumps creating large amounts of log spam
jcrespo added a project: Datacenter-Switchover-2018.jcrespo added subscribers: akosiaris, Joe.jcrespo added a comment. CC @Joe @akosiaris not to solve the specific issue, but to note it as a potential missed step/puppet code change related to switch datacenter. Having said that, if dumps are run from eqiad, they probably should query local dbs to prevent privacy issues? So technically we are running active-active if we have to account dumps? Will that break something?TASK DETAILhttps://phabricator.wikimedia.org/T204531EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Joe, akosiaris, hoo, Agabi10, Marostegui, ArielGlenn, Krinkle, fgiunchedi, jcrespo, jijiki, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, Volans, MGChecker, gnosygnu, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Blocker] T171928: Wikidata and dewiki databases locked
jcrespo changed the status of subtask T172489: Monitor read_only on all databases, make it page on masters from "Open" to "Stalled". TASK DETAILhttps://phabricator.wikimedia.org/T171928EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: gerritbot, mark, Marostegui, Elitre, Joe, jcrespo, greg, Mbch331, Smalyshev, MisterSynergy, TerraCodes, Jay8g, Lydia_Pintscher, Aklapper, Esc3300, PokestarFan, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, faidon, He7d3r, TheDJ, Jdforrester-WMF, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Project Column] T204838: Make Wikibase wb_terms term_search_key index the same as wb_term_search_key in WMF production
jcrespo moved this task from Triage to Blocked external/Not db team on the DBA board.jcrespo added a comment. ^I am assuming this doesn't have yet actionables for us (yet) just on the radar.TASK DETAILhttps://phabricator.wikimedia.org/T204838WORKBOARDhttps://phabricator.wikimedia.org/project/board/1060/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Aklapper, Marostegui, Addshore, Banyek, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production)
jcrespo added a comment. Maybe related or blocker: T106363 I am interested on this happening at some point, I just saw lots of revision data (not metadata) stored on the text table and its impact on a quick database recovery.TASK DETAILhttps://phabricator.wikimedia.org/T183490EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, JJMC89, Agabi10, Wikidata-bugs, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. We know the exact timestamps of missing rows, from db1071-bin.007238:795791989 2018-09-13 09:08:17 on the active (codfw) master: db2045-bin.005879:1036765620 to db1071-bin.007238:796727644 2018-09-13 09:58:26 on the active (codfw) master: db2045-bin.005880:132937482 We have recovered all sql code executed at that time, no there should not be permanent data loss, I am trying to recover those now in a way that will be safe do no further loss happens of newer transactions- this may take a bit, as this happened a long time ago. If this had been identified 3 days later, we could not even do that, as binlog is purged after 30 days. In the worse case scenario, we could publish those changes so they are redone, but I am going to try to apply them to the main affected tables (user, revision, page, text, ...) automatically. Content itself was not affected because content is stored through a different replication channel- so only the metadata needed to access it.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, Nandana, Banyek, jijiki, Mringgaard, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Volans, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. applied a fix to: db1109 db1071 db1104 db1101:3318 db1092 dbstore1002TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. We believe the most important issues (missing revisions, pages and users, and making accessible the content) has been fixed. We will lower the UBN after doing some additional sanity checks. Having said that, for the following days, some missing things like user preferences, what links here* pages and others may act strange. A longer term fix (reimport from a good state) will slowly be deployed in the following days.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a project: User-notice.jcrespo added a comment. For #user-notice, if it seems reasonable, see T206743#4658525 for the state of things- data was temporarily lost (for around 1 day) but then it was recovered once detected. There may be smaller issues (preferences lost, what links here, tags) that will disappear during the next week when a full-scale reimport gets applied. But at the moment, no data, revision or content should be lost. An incident report will be written to fully understand why this happened, as we know the what, but not what was the real root cause at the moment (skipping 50 minutes of data replication).TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Lowered Priority] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo lowered the priority of this task from "Unbreak Now!" to "High".jcrespo added a comment. High because data came back.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. @Pigsonthewing Please don't get worry- as I said at T206743#4658620 only primary data was recovered, cache tables may show still wrong data until we fully recover everything this week, showing some weird data. Please be patient, but don't touch anything that seems wrong, as it will only complicate the full recovery we are doing at the moment. Once the recovery is fully done, we will announce it here and you should have everything back- however if you edit or try to recover manually the edits, it will generate new revisions and that version will stay like that- as it will be the latest one (older versions will be stored in the history- and it may show as if you vandalized (removed content) from an article.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. @Addshore The main task now is to check pages and pages from revisions that were lost and then restored, and check edits done since last wednesday on those pages, as, while the data has not been lost, those may have put an edit over an older version as the latest, hiding some data. We need to generate a list of Qs to review.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. I'll generate a list once all slaves that serve mw traffic are re imaged :) This is true right now for all pooled read only replicas, but I need to fix the master.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T147169: Make sure Wikibase dump maintenance scripts solely use the "dump" db group
jcrespo added a comment. @jcrespo Do you have any more information for what to look out here? The problem is most likely the loadbalancer- working with old data, even if the server is not needed, generates lag warnings for some complex dependency of how lag check works (generating 40K logs per minute). Abandon ship- this is not going to work, and nothing that wikibase developers or DBAs do can fix it.TASK DETAILhttps://phabricator.wikimedia.org/T147169EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: Marostegui, gerritbot, Lucas_Werkmeister_WMDE, ArielGlenn, Addshore, aaron, Aklapper, Smalyshev, Lydia_Pintscher, jcrespo, aude, daniel, hoo, Nandana, Lahi, Gq86, GoranSMilovanovic, lisong, QZanden, LawExplorer, Jonas, Wikidata-bugs, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium
jcrespo added a comment. From another thread, from TimS: It sounds like the snapshot hosts were in fact needing and using the host. It's not LoadBalancer's fault if some maintenance script calls wfGetDB() without specifying a query group. Throwing an exception is the correct thing to do if the MW configuration is incorrect, since it allows the maintenance script to terminate and be restarted with different configuration.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, gnosygnu, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. After many fixes during the weekend, wb_terms also fixed on labs, all hosts should be consistent now, doing some extra checks to verify everything is good. @Addshore did you sent to wikidata users the list you compiled to check bot activity?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Lowered Priority] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo lowered the priority of this task from "High" to "Normal".jcrespo added a comment. We believe this to be fixed fully both wikireplicas and on production, but will not close until extra checks confirm so.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. @Addshore I thought you had communicated to wikidata users about that? Apparently not, or @Pigsonthewing didn't see it, could you link your messages to him?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. @Pigsonthewing I hope my comment at Wikidata Village Pump was helpful- if you think that is ok, I would suggest closing this task, and open a different one to track the merges of old history (this was to track the recovery from backups)?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Project Column] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo moved this task from In progress to Done on the DBA board.jcrespo added a comment. In particular, I see https://www.wikidata.org/wiki/Q2058295 properly merged.TASK DETAILhttps://phabricator.wikimedia.org/T206743WORKBOARDhttps://phabricator.wikimedia.org/project/board/1060/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T207006: Set wb_changes_dispatch ROW_FORMAT=COMPRESSED on install and update
jcrespo added a comment. loading a table with ROW_FORMAT=COMPRESSED but I am not sure compression is actually the cause, but maybe (I would actually be more sure about that) just a simple rebuild from scratch after a many edits are suffered (something common after >1000M edits). Compression is the way to go on Wikipedia no doubt, but I am unsure I would set it up for all Wikibase installations... specially smaller ones. We can talk more in depth in the future.TASK DETAILhttps://phabricator.wikimedia.org/T207006EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Aklapper, hoo, Addshore, Nandana, Banyek, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Minhnv-2809, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T208695: Duplicate key on several s8 replicas breaking replication
jcrespo added a comment. it got missed somehow How can that be?, the table was empty or clean at that time according to my notes-and this was verified by 2 people independently (Manuel and me) on 2 different servers as I have on my notes on the 16 oct, and later at T206743#4691048 , unless I am making a huge mistake or missing something. And even if it wasn't and somehow 2 people independently skipped it- the different edits added more than 1 comment, but apparently was 1 row was different? Could an interaction with archive create issues- eg. the undeletion of a revision so it gets moved from archive to revision and archive is know to been unreliable in the past?TASK DETAILhttps://phabricator.wikimedia.org/T208695EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jijiki, Anomie, Addshore, Banyek, Aklapper, jcrespo, Nandana, AndyTan, Imarlier, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Minhnv-2809, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T203709: Schema change for adding indexes of ct_tag_id
jcrespo added a comment. Could you check the list of schema changes and maintenance to be ran during switchover to test if they where also undone?TASK DETAILhttps://phabricator.wikimedia.org/T203709EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: jcrespo, Addshore, Stashbot, Marostegui, Tgr, Anomie, TTO, Catrope, Aklapper, gerritbot, Ladsgroup, CucyNoiD, Nandana, NebulousIris, Banyek, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, D3r1ck01, Jonas, MGChecker, Wikidata-bugs, aude, Dinoguy1000, Lydia_Pintscher, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T86530: Replace wb_terms table with more specialized mechanisms for terms (tracking)
jcrespo added a comment. To be fair, technically, this is resolved because wb_terms has been replaced, AFAIK, with a more specialized mechanism (several smaller and normalised tables) :-) TASK DETAIL https://phabricator.wikimedia.org/T86530 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Mholloway, Addshore, jcrespo, Ricordisamoa, Lydia_Pintscher, adrianheine, thiemowmde, Tobi_WMDE_SW, JeroenDeDauw, JanZerebecki, aude, Aklapper, Liuxinyu970226, daniel, hoo, Akuckartz, darthmon_wmde, Nickleh, Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Volker_E, Wikidata-bugs, GWicke, Dinoguy1000, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T165726: [Hackathon doc sprint] Improve deployment documentation
jcrespo closed subtask T165756: Create summary templates on Wikitech wiki to stop writing the same things everywhere, everytime as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T165726 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Aklapper, Dereckson, Akuckartz, apaskulin, CptViraj, Pavithraes, darthmon_wmde, Dibya, 94rain, DannyS712, Nandana, Tks4Fish, Cpaulf30, Lahi, Gq86, GoranSMilovanovic, Ivana_Isadora, Jayprakash12345, QZanden, Kizule, LawExplorer, Puik, DatGuy, Devwaker, Niklitov, _jensen, Urbanecm, rosalieper, xSavitar, Envlh, JEumerus, Scott_WUaS, Ananthsubray, Superzerocool, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, Wikidata-bugs, Snowolf, aude, Tobias1984, Dcljr, Dinoguy1000, Matanya, Mbch331, Rxy, Jay8g, Krenair, RhinosF1 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T103282: 30 second timeout on testRecentchanges connected to wikidata
jcrespo added a comment. @hoo you are right, but check my comment <https://phabricator.wikimedia.org/T101502#1341481> on a very similar issue on https://phabricator.wikimedia.org/T101502. We need to validate that change on all queries from the same piece of code (not only for this particular instance), on all server versions and all wikis, hopefully eliminating all FORCE INDEXes in the future. Actually, now that I see it, is this the same issue? TASK DETAIL https://phabricator.wikimedia.org/T103282 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Springle, aaron, aude, JanZerebecki, hoo, Lydia_Pintscher, greg, Aklapper, jayvdb, pywikibot-bugs-list, Wikidata-bugs, GWicke, Anomie, Krenair, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T102992: Review WikidataQuality DB schema
jcrespo added a subscriber: jcrespo. jcrespo added a comment. First comments from a MySQL DBA, assuming that goes to production: - Some tables lack of a `PRIMARY KEY`.** This should be a requirement for new tables**. - character strings do not have a specified `CHARACTER SET`. Check that those do not have to be `BINARY` fields for compatibility with current wiki fields or set them otherwise (by default, they may be restricted to latin characters in some systems). - Unless there is a reason not to, add the index definition along the table creation, and not with separate `CREATE INDEX` afterwards. While the results are the same from the point of view of the application, not from the point of view of maintaining hundreds of wikis. - Some `PRIMARY KEY`s use guid strings. Those should be justified by external dependencies, otherwise stick to int or bigint for better innodb performance. - `VARBINARY(15)` for import_date seems like a bad idea, there are specific data types for that. - There is a tendency of using `VARCHAR(255)` for variable length fields. Please note that using `VARCHAR(255)` is not an advantage over `VARCHAR(256)`, and that smaller sizes are usually preferred (again, unless justified by existing types). TASK DETAIL https://phabricator.wikimedia.org/T102992 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Tamslo, JAnstee_WMF, csteipp, Springle, hoo, Aklapper, daniel, Wikidata-bugs, aude, GWicke, Krenair, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T102992: Review WikidataQuality DB schema
jcrespo added a blocked task: T17441: Some tables lack unique or primary keys, may allow confusing duplicate data. TASK DETAIL https://phabricator.wikimedia.org/T102992 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Tamslo, JAnstee_WMF, csteipp, Springle, hoo, Aklapper, daniel, Wikidata-bugs, aude, GWicke, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T107602: Set up a public interface to the wikidata query service
jcrespo added a subscriber: jcrespo. TASK DETAIL https://phabricator.wikimedia.org/T107602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, jcrespo Cc: jcrespo, Legoktm, gerritbot, Smalyshev, BBlack, Joe, daniel, RobLa-WMF, Aklapper, aude, JanZerebecki, JeroenDeDauw, MrStradivarius, waldyrious, Krenair, MBlissett, bd808, Laddo, Addshore, Matanya, jkroll, Wikidata-bugs, Jdouglas, RobH, Manybubbles, mark, faidon, fgiunchedi, Dzahn, chasemp, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T99459: ips_site_page is too short to store some (full) page titles
jcrespo added a comment. By @hoo request, I have applied the schema change also to testwikidatawiki. Please, in the future, for us silly DBAs :-), we need affected wikis very explicitly. TASK DETAIL https://phabricator.wikimedia.org/T99459 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo, jcrespo Cc: gerritbot, Ladsgroup, Bene, MZMcBride, Krenair, jcrespo, Springle, Lydia_Pintscher, aude, daniel, Aklapper, hoo, Wikidata-bugs, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T107319: EntityUsageTable::touchUsages slow query
jcrespo added a subscriber: jcrespo. jcrespo added a comment. This query, which has to be executed in a serialized way due to replication, is causing a lag problem on some of the less powerful database slaves (db1047), and extra load on the rest of the servers. This is specially true for s3, where due to it containing hundreds of wikis, even with low traffic, it makes for most of its write load. Please note that even if this query is fast, it can create a huge bottleneck on the replication topology. Please consider executing it on a separate shard or storage system to minimize its impact to production. TASK DETAIL https://phabricator.wikimedia.org/T107319 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Aklapper, aude, Wikidata-bugs, Malyacko, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T107319: [Bug] EntityUsageTable::touchUsages slow query
jcrespo added a comment. @aude If you are increasing the priority based on my comment, don't. Regular databases can handle the extra load (for now), the main issue is in research databases or labs, where they contain all dbs, or when there is maintenance and we are in a degraded state. This currently is not a user-notice, but it is important enough to discuss a way to solve it. So the issue is that extra writes are not a problem for our masters -they are done concurrently-, but when we try to scale out, most of the writes are serialized on the logs, and there has been many times when there are lag problems that the current queries being executer are these. I do not thing the queries are slow- but I may be wrong, do not have statistics. You are right: let's get numbers first: how many are executed and how much time it takes (and for myself, I need to fully understand the model) before rushing into solutions (my suggestion was a bit premature). TASK DETAIL https://phabricator.wikimedia.org/T107319 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Aklapper, aude, Wikidata-bugs, Malyacko ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Raised Priority] T107319: [Bug] EntityUsageTable::touchUsages slow query
jcrespo raised the priority of this task from "High" to "Unbreak Now!". jcrespo added a subscriber: ori. jcrespo added a comment. Sorry, reverting back to Unbreak now, because this is breaking almost all wikis and the problems have been spoted now on production machines, not only on support machines: Executing things like this: UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20150619102700' WHERE eu_page_id = '1256862' AND ((eu_aspect = 'L.hu' AND eu_entity_id = 'Q1017251') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1017251') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1017287') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1017287') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1019531') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1019531') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1022151') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1022151') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1076731') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1076731') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1272631') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1272631') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1632441') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1632441') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1689740') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1689740') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1734119') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1734119') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1756917') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1756917') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1762083') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1762083') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1767147') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1767147') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q176874') OR (eu_aspect = 'T' AND eu_entity_id = 'Q176874') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1784043') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1784043') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1785657') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1785657') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1812906') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1812906') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1823175') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1823175') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1846197') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1846197') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1937519') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1937519') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1938332') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1938332') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1957359') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1957359') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1969100') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1969100') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1970044') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1970044') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q1996076') OR (eu_aspect = 'T' AND eu_entity_id = 'Q1996076') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q2016409') OR (eu_aspect = 'T' AND eu_entity_id = 'Q2016409') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q2020661') OR (eu_aspect = 'T' AND eu_entity_id = 'Q2020661') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q2034240') OR (eu_aspect = 'T' AND eu_entity_id = 'Q2034240') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q2036571') OR (eu_aspect = 'T' AND eu_entity_id = 'Q2036571') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q2050445') OR (eu_aspect = 'T' AND eu_entity_id = 'Q2050445') OR (eu_aspect = 'L.hu' AND eu_entity_id = 'Q2065507') OR
[Wikidata-bugs] [Maniphest] [Lowered Priority] T107319: [Bug] EntityUsageTable::touchUsages slow query
jcrespo lowered the priority of this task from "Unbreak Now!" to "High". jcrespo added a comment. This is the lag report of db1028. F1511025: Screenshot from 2015-08-15 11:30:15.png <https://phabricator.wikimedia.org/F1511025> Seems to have calmed for now. TASK DETAIL https://phabricator.wikimedia.org/T107319 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: JanZerebecki, daniel, hoo, Addshore, Lydia_Pintscher, ori, jcrespo, Aklapper, aude, Wikidata-bugs, Malyacko ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T107319: [Bug] EntityUsageTable::touchUsages slow query
jcrespo added a comment. And here it is actual data of an example spike: Host UserSchema Client Source Thread Transaction Runtime Stamp db1038wikiusersimplewiki mw1001 - 5096510924 30A18B0D6 15s 2015-08-15 10:04:05 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20140130031809' WHERE eu_page_id = '232335' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiusermwlwiki mw1006 - 5096510302 30A18AA42 14s 2015-08-15 10:03:55 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20130201200841' WHERE eu_page_id = '1' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiuserskwiki mw1009 - 5096509493 30A18A10F 13s 2015-08-15 10:03:35 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20150218224016' WHERE eu_page_id = '1' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiuserelwiki mw1016 - 5096509480 30A18A0F7 13s 2015-08-15 10:03:35 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20150427204519' WHERE eu_page_id = '260239' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiuserkawiki mw1015 - 5096509531 30A18A156 12s 2015-08-15 10:03:35 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20150803200253' WHERE eu_page_id = '16312' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiuserguwiki mw1005 - 5096509757 30A18A40B 12s 2015-08-15 10:03:55 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20150815100353' WHERE eu_page_id = '1' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiusertlwiki mw1007 - 5096509558 30A18A19E 11s 2015-08-15 10:03:35 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20141113113613' WHERE eu_page_id = '1' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiusersimplewiki mw1012 - 5096509479 30A18A0F6 11s 2015-08-15 10:03:35 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20150815100333' WHERE eu_page_id = '232335' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiuserbewiki mw1013 localhost 5096509524 30A18A14A 11s 2015-08-15 10:03:45 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages 127.0.0.1 */ `wbc_entity_usage` SET eu_touched = '20150815100343' WHERE eu_page_id = '50792' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiuserbnwiki mw1008 - 5096509501 30A18A11E 11s 2015-08-15 10:03:45 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20150815100343' WHERE eu_page_id = '796' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiusersrwiki mw1008 localhost 5096509510 30A18A12D 9s 2015-08-15 10:03:45 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages 127.0.0.1 */ `wbc_entity_usage` SET eu_touched = '20150815100335' WHERE eu_page_id = '196554' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiusersdwiki mw1014 - 5096509431 30A18A000 9s 2015-08-15 10:03:45 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20150815100335' WHERE eu_page_id = '2238' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiuserhiwiki mw1013 - 5096510664 30A18AE18 9s 2015-08-15 10:04:05 UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages */ `wbc_entity_usage` SET eu_touched = '20150520065035' WHERE eu_page_id = '220108' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q5296')) db1038wikiuserdawiki mw1008 - 5096510405 30A18AB5C 9s 2015-08
[Wikidata-bugs] [Maniphest] [Commented On] T107319: [Bug] EntityUsageTable::touchUsages slow query
jcrespo added a comment.
Could this job have caused:
/* JobRunner::commitMasterChanges 127.0.0.1 */
GET_LOCK('jobrunner-serial-commit', 30) AS lockstatus /*
31bdf25cc42f95f4f4a9b493cafe0299 db1024 plwiki 11s */ /* localhost */
we had another large lag spike around the same time than other touchusages, but
it could have just been a coincidence? This other "query" caused 7000 hits with
an avg waiting time of 15 seconds, causing lag on s2. If not, I will open
another ticket for this other issue.
Another proof that touchUsages may be too aaggressive is that sometimes it
conflicts with itself. Updating by primary key, as suggested, should solve this.
Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages 10.64.16.13 1205 Lock
wait timeout exceeded; try restarting transaction (10.64.16.13) UPDATE
`wbc_entity_usage` SET eu_touched = '20150822101128' WHERE eu_page_id =
'616905' AND ((eu_aspect = 'S' AND eu_entity_id = 'Q14484217'))
BTW, thank you for looking at this- you have been a really responsive team, and
this is really appreciated.
TASK DETAIL
https://phabricator.wikimedia.org/T107319
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: gerritbot, JanZerebecki, daniel, hoo, Addshore, Lydia_Pintscher, ori,
jcrespo, Aklapper, aude, Wikidata-bugs, Malyacko
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T107319: [Bug] EntityUsageTable::touchUsages slow query
jcrespo added a comment. I am almost sure that the first part of my previous comment was not caused by this job, but a very large LinksUpdate::incrTableUpdate, sorry about that. TASK DETAIL https://phabricator.wikimedia.org/T107319 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: gerritbot, JanZerebecki, daniel, hoo, Addshore, Lydia_Pintscher, ori, jcrespo, Aklapper, aude, Wikidata-bugs, Malyacko ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T107319: [Bug] EntityUsageTable::touchUsages slow query
jcrespo added a comment. @JanZerebecki not sure if you are asking me: Yes, IF that is the case (I cannot say)- batching will actually be worse than no batching at all- as it will be slower than a single query and more open to clashing with other queries without any of the advantages (reduced lag). TASK DETAIL https://phabricator.wikimedia.org/T107319 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: gerritbot, JanZerebecki, daniel, hoo, Addshore, Lydia_Pintscher, ori, jcrespo, Aklapper, aude, Wikidata-bugs, Malyacko ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T111535: Wikibase\Repo\Store\SQL\EntityPerPageTable::{closure} creating high number of deadlocks
jcrespo created this task.
jcrespo added a subscriber: jcrespo.
jcrespo added projects: Wikidata, MediaWiki-Database.
Herald added a subscriber: Aklapper.
TASK DESCRIPTION
There seems to be a large number of API calls to wikidata creating a high
number of deadlocks like this (450/hour):
{
"_index": "logstash-2015.09.04",
"_type": "mediawiki",
"_id": "AU-ZFLBnIkWKhegMyoko",
"_score": null,
"_source": {
"message":
"Wikibase\\Repo\\Store\\SQL\\EntityPerPageTable::{closure}\t10.64.32.28\t1213\tDeadlock
found when trying to get lock; try restarting transaction
(10.64.32.28)\tINSERT INTO `wb_entity_per_page`
(epp_entity_id,epp_entity_type,epp_page_id,epp_redirect_target) VALUES
('16507538','item','22922124',NULL)",
"@version": 1,
"@timestamp": "2015-09-04T15:58:33.340Z",
"type": "mediawiki",
"host": "mw1221",
"level": "ERROR",
"tags": [
"syslog",
"es",
"es",
"normalized_message_trimmed"
],
"channel": "wfLogDBError",
"url": "/w/api.php",
"ip": "10.64.32.107",
"http_method": "POST",
"server": "www.wikidata.org",
"referrer": null,
"uid": "4943efb",
"process_id": 21645,
"wiki": "wikidatawiki",
"db_server": "10.64.32.28",
"db_name": "wikidatawiki",
"db_user": "wikiuser",
"method": "DatabaseBase::reportQueryError",
"errno": 1213,
"error": "Deadlock found when trying to get lock; try restarting
transaction (10.64.32.28)",
"sql1line": "INSERT INTO `wb_entity_per_page`
(epp_entity_id,epp_entity_type,epp_page_id,epp_redirect_target) VALUES
('16507538','item','22922124',NULL)",
"fname": "Wikibase\\Repo\\Store\\SQL\\EntityPerPageTable::{closure}",
"normalized_message":
"Wikibase\\Repo\\Store\\SQL\\EntityPerPageTable::{closure}\t10.64.32.28\t1213\tDeadlock
found when trying to get lock; try restarting transaction
(10.64.32.28)\tINSERT INTO `wb_entity_per_page`
(epp_entity_id,epp_entity_type,epp_page_id,epp_redirect_target) VALUE"
},
"sort": [
1441382313340
]
}
They started on Friday 4 Sept at around 10:50 and continues 5 hours later.
They are not causing issues on the infrastructure, but they seem to be
interfering with themselves. We need to identify the cause: if it is legitimate
traffic, change how the inserts are done; if it is abnormal traffic, maybe
avoid it in the first place.
TASK DETAIL
https://phabricator.wikimedia.org/T111535
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Aklapper, jcrespo, Wikidata-bugs, aude
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Declined] T111535: Wikibase\Repo\Store\SQL\EntityPerPageTable::{closure} creating high number of deadlocks
jcrespo closed this task as "Declined". jcrespo claimed this task. jcrespo added a comment. I'm more than ok with the work you are doing, I was just suggesting making the code faster :-) TASK DETAIL https://phabricator.wikimedia.org/T111535 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: JanZerebecki, Addshore, Aklapper, jcrespo, Wikidata-bugs, aude ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T111769: EntityUsageTable::touchUsageBatch slow query
jcrespo added a subscriber: jcrespo. TASK DETAIL https://phabricator.wikimedia.org/T111769 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, aude, Aklapper, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T111535: Wikibase\Repo\Store\SQL\EntityPerPageTable::{closure} creating high number of deadlocks
jcrespo added a blocked task: T30599: Deadlock tracking bug (tracking). TASK DETAIL https://phabricator.wikimedia.org/T111535 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: aude, JanZerebecki, Addshore, Aklapper, jcrespo, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T111535: Wikibase\Repo\Store\SQL\EntityPerPageTable::{closure} creating high number of deadlocks
jcrespo triaged this task as "Low" priority. TASK DETAIL https://phabricator.wikimedia.org/T111535 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: aude, JanZerebecki, Addshore, Aklapper, jcrespo, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T111535: Wikibase\Repo\Store\SQL\EntityPerPageTable::{closure} creating high number of deadlocks
jcrespo added a comment. I set the priority to low "nice to fix it, but it is not breaking anything now". Feel free to disagree with me. TASK DETAIL https://phabricator.wikimedia.org/T111535 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: aude, JanZerebecki, Addshore, Aklapper, jcrespo, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T111954: Origin 'http://...' is therefore not allowed access to query.wikidata.org
jcrespo closed blocking task Restricted Task as "Invalid". TASK DETAIL https://phabricator.wikimedia.org/T111954 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Smalyshev, Karima, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, JanZerebecki ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T111954: Origin 'http://...' is therefore not allowed access to query.wikidata.org
jcrespo removed a blocking task: Restricted Task. TASK DETAIL https://phabricator.wikimedia.org/T111954 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Smalyshev, Karima, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, JanZerebecki ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T111769: EntityUsageTable::touchUsageBatch slow query
jcrespo added a comment. It could be an enqueue issue, if thousands of those queries are sent at the same time (due to bot traffic, for example), they would be enqueued on server side and executed with controlled concurrency. I could check the binary logs to see if that is the case, and profile the database server when that happens (although I would need to know on which server/wiki that happens, and catch it in the moment). We recently had issues with bots doing legit but heavier-than-usual traffic on wikidata, could be related. TASK DETAIL https://phabricator.wikimedia.org/T111769 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: daniel, jcrespo, aude, Aklapper, Wikidata-bugs, GWicke, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
