[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-12-12T08:37:00Z] Remove old backup directory from db1116 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, StashbotCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, _jensen, D3r1ck01, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. If there is no space issues on that host, there is no harm in leaving it a bit longer I would say.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, _jensen, D3r1ck01, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Banyek added a comment. on db1116 we still have the backup dir: drwxr-xr-x 8 mysql mysql 8.0K Oct 11 13:21 sqldata.s8_BACKUP_T206743 I guess we shall it remove now, as this ticket is resolved (and we still remember the issue) ?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, BanyekCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, _jensen, D3r1ck01, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. @Pigsonthewing I hope my comment at Wikidata Village Pump was helpful- if you think that is ok, I would suggest closing this task, and open a different one to track the merges of old history (this was to track the recovery from backups)?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. In T206743#4691328, @jcrespo wrote: @Addshore I thought you had communicated to wikidata users about that? Apparently not, or @Pigsonthewing didn't see it, could you link your messages to him? I'll leave that to @Lea_Lacroix_WMDE. This is still on the needs announcement column for us. The related pages that I created can be found @ https://www.wikidata.org/wiki/User:Addshore/2018/10/DC_Switch_IssueTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, AddshoreCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. @Addshore I thought you had communicated to wikidata users about that? Apparently not, or @Pigsonthewing didn't see it, could you link your messages to him?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. I guess it is because what @Addshore described at T206743#4662054 ?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. Final round of tables that have been checked and are clean: abuse_filter_action abuse_filter_history betafeatures_user_counts content content_models global_block_whitelist ipblocks math mathoid ores_model protected_titles querycache_info revision_comment_temp securepoll_elections securepoll_entity securepoll_lists securepoll_msgs securepoll_voters securepoll_properties sites site_identifiers site_stats slot_roles templatelinks translate_metadata user_former_groups user_groups At this point, all the tables in s8 have been checked and confirmed clean.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. The following tables are empty so no need to check: actor config filearchive filejournal globalblocks hidden image image_comment_temp interwiki ipblocks_restrictions job l10n_cache oldimage revision_actor_temp searchindex securepoll_cookie_match securepoll_options securepoll_questions securepoll_strike securepoll_votes transcache transcode updatelog uploadstashTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. wb_terms is finished and it is all clean.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. More tables to check after the ones checked at: T206743#4685983 abuse_filter_log (T206743#4685983) was fixed by Jaime (I have rechecked again and it is indeed now consistent). The following tables are clean between db1087 and db1092: abuse_filter category change_tag_def cu_changes externallinks geo_tags imagelinks iwlinks linter log_search ores_classification page page_props page_restrictions pagelinks spoofuser tag_summary templatelinks user user_newtalk user_properties watchlist wb_changes_subscription wb_items_per_site wbc_entity_usage The following tables reported differences and have already been fixed by Jaime (and confirmed after a second check by me) change_tag On-going check for the last table (which takes several hours: wb_termsTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-23T06:39:09Z] Stop replication on db1092 and db1087 for checking T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. These tables are fine: logging revision slots text Starting to check: abuse_filter category change_tag change_tag_def cu_changes externallinks geo_tags imagelinks ip_changes iwlinks linter log_search ores_classification page page_props page_restrictions pagelinks spoofuser tag_summary templatelinks user user_newtalk user_properties watchlist wb_changes_subscription wb_items_per_site wbc_entity_usageTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. In T206743#4685334, @jcrespo wrote: @Addshore did you sent to wikidata users the list you compiled to check bot activity? YesTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. The following tables have been re-checked on db1092 (recloned from codfw) vs db1087 (manually fixed): abuse_filter_log archive babel bv2013_edits bv2015_edits bv2017_edits categorylinks comment cu_log ipblocks langlinks module_deps object_cache querycache querycachetwo recentchanges redirect revtag translate_reviews translate_groupreviews translate_groupstats translate_messageindex translate_sections valid_tag wb_changes wb_changes_dispatch wb_id_counters wb_property_info wbqc_constraints wbs_propertypairs On-going: logging revision slots text Found differences on: abuse_filter_log which has 66 rows more on db1092 than on db1087 (and sanitarium, and labs) and db1071 (master).TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. After many fixes during the weekend, wb_terms also fixed on labs, all hosts should be consistent now, doing some extra checks to verify everything is good. @Addshore did you sent to wikidata users the list you compiled to check bot activity?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. Update: pagelinks is now being re-imported on labs (this table is around 136G) so it will take a whileTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. Update from Jaime 18th Oct 16:05: s8 core hosts all finished getting fixed (pending labs)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. In T206743#4676466, @ArielGlenn wrote: In T206743#4675755, @Banyek wrote: on db1124 with instance s8 we have a repliation error as Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db1087-bin.003073, end_log_pos 582738698 Where was this error message logged? I have been looking for an error log on db1124 without success. Probably on show slave status, but that error was fixed already. This host is sanitarium, so it is not user facing error (it does create lag on labs though). This host is expected to fail while it is getting fixed. Right now there is another error there, as replication got broken again.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
ArielGlenn added a comment. In T206743#4675755, @Banyek wrote: on db1124 with instance s8 we have a repliation error as Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db1087-bin.003073, end_log_pos 582738698 Where was this error message logged? I have been looking for an error log on db1124 without success.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, ArielGlennCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. Update from 17th at 19:04 from Jaime: all tables except wb_terms, which is half done, should be equal on the s8 masterTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Banyek added a comment. on db1124 with instance s8 we have a repliation error as Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db1087-bin.003073, end_log_pos 582738698TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, BanyekCc: Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. Update from Jaime at 12:10 UTC All tables on s8 master fixed except pagelinks and wb_termsTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-17T06:52:01Z] fixing s8 master drifts T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. Update from yesterday at around 14:00UTC @jcrespo has done an amazing job of manually checking and fixing most of the tables on db1087 (which is the labs master, and it is not as easy to reclone as the others). He's gone thru all the tables to check and fix them. Right all the tables apart from pagelinks and wb_terms are supposed to be fixed (although replication is broken on db1124 sanitarium master for wb_items_per_site, which is kind of expected with such amount of rows to check - we are looking at it). pagelinks and wb_terms are huge tables, and they cannot be diffed at once because our mysql client reaches OOM, so these checks need to be split on smaller diffs, that's why it is taking a bit longer. Thanks Jaime for this huge work.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-16T06:05:03Z] stopping db1092 and db1087 in sync T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. I have prepared https://www.wikidata.org/wiki/User:Addshore/2018/10/DC_Switch_Issue which includes the pages that will need to be reviewed.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. So as Jaime said at T206743#4666169 all the pooled replicas in eqiad have now the content from codfw, so that is now consistent. What is pending is db1087 (depooled) which is the master for db1124 (labsdb master) and labsdb, and of course the master, db1071 which doesn't receive reads. So labsdb still have inconsistencies. We are now working on fixing those, db1087 which will automatically (via replication) will fix db1124 and labs.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-15T13:16:48Z] stopping db1092 and db1087 in sync T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. I'll generate a list once all slaves that serve mw traffic are re imaged :) This is true right now for all pooled read only replicas, but I need to fix the master.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. In T206743#4665661, @jcrespo wrote: @Addshore The main task now is to check pages and pages from revisions that were lost and then restored, and check edits done since last wednesday on those pages, as, while the data has not been lost, those may have put an edit over an older version as the latest, hiding some data. We need to generate a list of Qs to review. I'll generate a list once all slaves that serve mw traffic are re imaged :)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-15T09:46:16Z] Synchronized wmf-config/db-eqiad.php: Depool db1092 for recloning - T206743 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. @Addshore The main task now is to check pages and pages from revisions that were lost and then restored, and check edits done since last wednesday on those pages, as, while the data has not been lost, those may have put an edit over an older version as the latest, hiding some data. We need to generate a list of Qs to review.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-15T08:46:38Z] Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 - T206743 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467285 merged by jenkins-bot: [operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1104 https://gerrit.wikimedia.org/r/467285TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. I wrote a little crappy but effective query that I can just run on all of the shards to see if anything else was affected. select max(rev_id), min(rev_id), count(*), max(rev_id) - min(rev_id) as diff from ( select rev_id, rev_timestamp from revision order by rev_id DESC limit 2000 ) as something where rev_timestamp > 20180913090816 and rev_timestamp < 20180913095825; It shows us what happened to the rev_id in the 50 minute period that we are concerned with. Small difference could be put down to revdel etc, as long as we don't see anything that is near 0 everything should be fine. S1 - enwiki +-+-+--+--+ | max(rev_id) | min(rev_id) | count(*) | diff | +-+-+--+--+ | 859332479 | 859327787 | 4604 | 4692 | +-+-+--+--+ S2 - ptwiki +-+-+--+--+ | max(rev_id) | min(rev_id) | count(*) | diff | +-+-+--+--+ |53121516 |53121457 | 58 | 59 | +-+-+--+--+ S3 - afwiki +-+-+--+--+ | max(rev_id) | min(rev_id) | count(*) | diff | +-+-+--+--+ | 1814203 | 1814188 | 16 | 15 | +-+-+--+--+ S4 - commonswiki +-+-+--+--+ | max(rev_id) | min(rev_id) | count(*) | diff | +-+-+--+--+ | 319952253 | 319947702 | 4387 | 4551 | +-+-+--+--+ S5 - dewiki +-+-+--+--+ | max(rev_id) | min(rev_id) | count(*) | diff | +-+-+--+--+ | 180869513 | 180868285 | 1188 | 1228 | +-+-+--+--+ S6 - frwiki +-+-+--+--+ | max(rev_id) | min(rev_id) | count(*) | diff | +-+-+--+--+ | 152158909 | 152157584 | 1301 | 1325 | +-+-+--+--+ S7 - arwiki +-+-+--+--+ | max(rev_id) | min(rev_id) | count(*) | diff | +-+-+--+--+ |30515544 |30515320 | 220 | 224 | +-+-+--+--+ So it looks like s8 was indeed the only issue @MarosteguiTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. @Pigsonthewing Please don't get worry- as I said at T206743#4658620 only primary data was recovered, cache tables may show still wrong data until we fully recover everything this week, showing some weird data. Please be patient, but don't touch anything that seems wrong, as it will only complicate the full recovery we are doing at the moment. Once the recovery is fully done, we will announce it here and you should have everything back- however if you edit or try to recover manually the edits, it will generate new revisions and that version will stay like that- as it will be the latest one (older versions will be stored in the history- and it may show as if you vandalized (removed content) from an article.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467285 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1104 https://gerrit.wikimedia.org/r/467285TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-15T07:24:08Z] Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 48s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-15T07:12:16Z] Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467261 merged by jenkins-bot: [operations/mediawiki-config@master] db-eqiad.php: Depool db1104 https://gerrit.wikimedia.org/r/467261TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467261 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Depool db1104 https://gerrit.wikimedia.org/r/467261TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467260 merged by jenkins-bot: [operations/mediawiki-config@master] db-eqiad.php: Fully repool db1109 https://gerrit.wikimedia.org/r/467260TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467260 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Fully repool db1109 https://gerrit.wikimedia.org/r/467260TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467252 merged by jenkins-bot: [operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1109 https://gerrit.wikimedia.org/r/467252TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467252 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1109 https://gerrit.wikimedia.org/r/467252TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-15T05:16:12Z] Stop MySQL on db1109 for recloning - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467249 merged by jenkins-bot: [operations/mediawiki-config@master] db-eqiad.php: Depool db1109 https://gerrit.wikimedia.org/r/467249TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 467249 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Depool db1109 https://gerrit.wikimedia.org/r/467249TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Pigsonthewing added a comment. The item I gave as an example above still shows the wrong image: https://www.wikidata.org/wiki/Q2058295 It is also missing other data. Note this diff, the most recent edit (as I write): https://www.wikidata.org/w/index.php?title=Q2058295&diff=762891812&oldid=745457545 as it was made with Mix'n'Match, only one ID should have been added.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, PigsonthewingCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. Here is a list of all pages that will likely be affected already with data being lost in the revisions that have since been restored: MariaDB [wikidatawiki]> SELECT DISTINCT rev_page, page_title, page_namespace, page_touched -> FROM revision AS revision -> INNER JOIN page AS page ON revision.rev_page = page.page_id -> WHERE rev_id > 745452473 and rev_id < 745468910 and page_touched > 20181010151500 -> ORDER BY page_namespace, page_title; +--+-+++ | rev_page | page_title | page_namespace | page_touched | +--+-+++ | 957791 | Q1008351| 0 | 20181011183506 | | 972072 | Q1022718| 0 | 2018102403 | | 11573912 | Q10269330 | 0 | 20181011102631 | | 11618130 | Q10313336 | 0 | 20181011183306 | | 12075277 | Q10801324 | 0 | 20181011231428 | | 1037514 | Q1089690| 0 | 20181011224900 | | 115186 | Q112562 | 0 | 20181011232641 | | 1072735 | Q1125675| 0 | 2018102401 | | 1088182 | Q1141499| 0 | 20181011093245 | | 1098210 | Q1151801| 0 | 2018102357 | | 1103257 | Q1156971| 0 | 20181011231351 | | 1140371 | Q1196895| 0 | 20181011150707 | | 130504 | Q128504 | 0 | 20181012061613 | | 1246692 | Q1306726| 0 | 20181012061648 | | 1278081 | Q1338401| 0 | 20181011010159 | | 1304828 | Q1365309| 0 | 2018102351 | | 140372 | Q138671 | 0 | 20181012114338 | | 140517 | Q138820 | 0 | 20181011032316 | | 1373695 | Q1435223| 0 | 20181011093244 | | 1444721 | Q1507379| 0 | 2018102349 | | 17026621 | Q15378860 | 0 | 20181011195823 | | 1480008 | Q1543446| 0 | 2018102346 | | 17087585 | Q15443105 | 0 | 2018102023 | | 17343834 | Q15708744 | 0 | 20181012023419 | | 17825650 | Q16199113 | 0 | 20181010184808 | | 17865127 | Q16240975 | 0 | 20181012124819 | | 1575620 | Q1641483| 0 | 2018102343 | | 1607340 | Q1673849| 0 | 20181011200423 | | 168071 | Q167548 | 0 | 2018102132 | | 1611743 | Q1678281| 0 | 2018102340 | | 1616952 | Q1683527| 0 | 20181011102520 | | 1624134 | Q1690779| 0 | 2018102338
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. root@neodymium:/home/marostegui# ./section s8 | while read host port; do echo $host; mysql.py -h$host:$port wikidatawiki -BN -e "select page_id, page_title, page_latest from page where page_id = 99480;";done labsdb1011.eqiad.wmnet 99480 Q97215 745463455 labsdb1010.eqiad.wmnet 99480 Q97215 745463455 labsdb1009.eqiad.wmnet 99480 Q97215 745463455 dbstore2001.codfw.wmnet 99480 Q97215 745463455 dbstore1002.eqiad.wmnet 99480 Q97215 745463455 db2094.codfw.wmnet 99480 Q97215 745463455 db2086.codfw.wmnet 99480 Q97215 745463455 db2085.codfw.wmnet 99480 Q97215 745463455 db2083.codfw.wmnet 99480 Q97215 745463455 db2082.codfw.wmnet 99480 Q97215 745463455 db2081.codfw.wmnet 99480 Q97215 745463455 db2080.codfw.wmnet 99480 Q97215 745463455 db2079.codfw.wmnet 99480 Q97215 745463455 db2045.codfw.wmnet 99480 Q97215 745463455 db1124.eqiad.wmnet 99480 Q97215 745463455 db1116.eqiad.wmnet 99480 Q97215 745463455 db1109.eqiad.wmnet 99480 Q97215 745463455 db1104.eqiad.wmnet 99480 Q97215 745463455 db1101.eqiad.wmnet 99480 Q97215 745463455 db1099.eqiad.wmnet 99480 Q97215 745463455 db1092.eqiad.wmnet 99480 Q97215 745463455 db1087.eqiad.wmnet 99480 Q97215 745463455 db1071.eqiad.wmnet 99480 Q97215 745463455TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. Using the example that I had in T206743#4661943 the pages that were fixable should now be fixed MariaDB [wikidatawiki]> select page_id, page_title, page_latest from page where page_id = 99480; +-++-+ | page_id | page_title | page_latest | +-++-+ | 99480 | Q97215 | 745463455 | +-++-+ 1 row in set (0.00 sec) MariaDB [wikidatawiki]> select rev_id from revision where rev_page = 99480 order by rev_id desc limit 2; +---+ | rev_id| +---+ | 745463455 | | 737776142 | +---+ 2 rows in set (0.00 sec) @Marostegui I guess you could confirm that using the powers you demonstrated in T206743#4662022 ? :)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-12T18:37:06Z] modified attachLatest.php script finished running over 9395 pages T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. In T206743#4662033, @Marostegui wrote: In T206743#4662023, @Addshore wrote: I'll run a maint script to fix the page_latest of the 9000ish pages that have an incorrect one so that we don't end up with more inconsistencies / broken stuff moving forward. But what will happen with the correct data we have in codfw?? Well after the script run both DCs should be correct. Essentially what has happened is: Switch from codfw to eqiad, resulting in the missing transactions One of the pages that had missing updates was edited during that time, making a new revision (with a newer rev id) and bumping the page_latest When we then added the transactions back, the previous updates to the page table changes the page_latest back to an ID prior to what is currently the latest. Once the script has finished the fallout of this whole thing should be that there are some inconsistencies with the pages / history for the pages listed in T206743#4661906 which were edited during this odd time between the switch back to eqiad and the adding of these revisions / running of this script.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. In T206743#4662023, @Addshore wrote: I'll run a maint script to fix the page_latest of the 9000ish pages that have an incorrect one so that we don't end up with more inconsistencies / broken stuff moving forward. But what will happen with the correct data we have in codfw??TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-12T18:25:26Z] running modified attachLatest.php script over ~9000 pages on wikidatawiki (with added wait for slaves) T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. I'll run a maint script to fix the page_latest of the 9000ish pages that have an incorrect one so that we don't end up with more inconsistencies / broken stuff moving forward.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. In T206743#4661943, @Addshore wrote: Which means there are ~9433 pages that now probably have the wrong page_latest, for example: We are fixing that by recloning the hosts (T206743#4658362) from codfw, which have the correct data: See how db1099 and db1101 already have the good data. root@neodymium:/home/marostegui# ./section s8 | while read host port; do echo $host; mysql.py -h$host:$port wikidatawiki -BN -e "select page_id, page_title, page_latest from page where page_id = 99480;";done labsdb1011.eqiad.wmnet 99480 Q97215 737776142 labsdb1010.eqiad.wmnet 99480 Q97215 737776142 labsdb1009.eqiad.wmnet 99480 Q97215 737776142 dbstore2001.codfw.wmnet 99480 Q97215 745463455 dbstore1002.eqiad.wmnet 99480 Q97215 737776142 db2094.codfw.wmnet 99480 Q97215 745463455 db2086.codfw.wmnet 99480 Q97215 745463455 db2085.codfw.wmnet 99480 Q97215 745463455 db2083.codfw.wmnet 99480 Q97215 745463455 db2082.codfw.wmnet 99480 Q97215 745463455 db2081.codfw.wmnet 99480 Q97215 745463455 db2080.codfw.wmnet 99480 Q97215 745463455 db2079.codfw.wmnet 99480 Q97215 745463455 db2045.codfw.wmnet 99480 Q97215 745463455 db1124.eqiad.wmnet 99480 Q97215 737776142 db1116.eqiad.wmnet 99480 Q97215 745463455 db1109.eqiad.wmnet 99480 Q97215 737776142 db1104.eqiad.wmnet 99480 Q97215 737776142 db1101.eqiad.wmnet 99480 Q97215 745463455 db1099.eqiad.wmnet 99480 Q97215 745463455 db1092.eqiad.wmnet 99480 Q97215 737776142 db1087.eqiad.wmnet 99480 Q97215 737776142 db1071.eqiad.wmnet 99480 Q97215 737776142 The idea with the quick was to get some rows in place, rather than have no rows.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. We are also fully restoring the eqiad hosts from codfw which has all the fine data. The fix done on Friday was a quick fix to get all the data in there. On eqiad, db1099 and db1101 should already have the proper data. They were recloned from codfwTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. It looks like we could fix these with the attachLatest.php maint script but it would need a little bit of modification.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. Which means there are ~9433 pages that now probably have the wrong page_latest, for example: MariaDB [wikidatawiki]> select page_id, page_title, page_latest from page where page_id = 99480; +-++-+ | page_id | page_title | page_latest | +-++-+ | 99480 | Q97215 | 737776142 | +-++-+ 1 row in set (0.00 sec) MariaDB [wikidatawiki]> select rev_id from revision where rev_page = 99480 order by rev_id desc limit 2; +---+ | rev_id| +---+ | 745463455 | | 737776142 | +---+ 2 rows in set (0.00 sec) These pages can be found with the following query: SELECT DISTINCT rev_page, page_title, page_namespace, page_touched FROM revision AS revision INNER JOIN page AS page ON revision.rev_page = page.page_id WHERE rev_id > 745452473 and rev_id < 745468910 and page_touched < 20181010151500 ORDER BY page_namespace, page_title;TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. In T206743#4661704, @Pigsonthewing wrote: I'm pretty sure I added an image: https://commons.wikimedia.org/wiki/File:Sandy_Wollaston_(cropped).jpg to: https://www.wikidata.org/wiki/Q2058295 that day (though of course its possible I was distracted and didn't complete the task); it's not there now. I see it in the revision history. https://www.wikidata.org/w/index.php?diff=745457425&oldid=744814394&title=Q2058295 But the current parsed page does indeed show the wrong image. F26560208: image.png It looks like an edit was made and the wrong latest revision was chosen from the page table. This means the edit was patched onto the version of the entity before the edits that were made during the 50 minute slot that was lost. F26560328: image.png It looks like we should write something to grab a list of pages that were affected and try to fix this.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Pigsonthewing added a comment. I'm pretty sure I added an image: https://commons.wikimedia.org/wiki/File:Sandy_Wollaston_(cropped).jpg to: https://www.wikidata.org/wiki/Q2058295 that day (though of course its possible I was distracted and didn't complete the task); it's not there now.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, PigsonthewingCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Addshore added a comment. In T206743#4661592, @Lea_Lacroix_WMDE wrote: User:Jason.nlw told me that during the effected period, he was in the middle of a batch upload, and some data was not restored yet. https://www.wikidata.org/w/index.php?title=Special:Contributions&offset=20180913095817&limit=500&contribs=user&target=Jason.nlw&namespace=&tagfilter=&start=2018-09-13&end=2018-09-13 Can someone have a look? Thanks. It looks like all of the revisions for that user exist in both data centers: EQIAD: MariaDB [wikidatawiki]> select count(*) from revision where rev_user = ( SELECT user_id from user where user_name = 'Jason.nlw' ); +--+ | count(*) | +--+ | 244293 | +--+ 1 row in set (0.14 sec) CODFW: MariaDB [wikidatawiki]> select count(*) from revision where rev_user = ( SELECT user_id from user where user_name = 'Jason.nlw' ); +--+ | count(*) | +--+ | 244293 | +--+ 1 row in set (0.21 sec) Do you have any other details? It could be that the user was just doing the batch upload during the data centre switch over period, at which time wikidata was in read only mode for a few minutes.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Lea_Lacroix_WMDE added a comment. User:Jason.nlw told me that during the effected period, he was in the middle of a batch upload, and some data was not restored yet. https://www.wikidata.org/w/index.php?title=Special:Contributions&offset=20180913095817&limit=500&contribs=user&target=Jason.nlw&namespace=&tagfilter=&start=2018-09-13&end=2018-09-13 Can someone have a look? Thanks.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, Lea_Lacroix_WMDECc: Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 466710 merged by jenkins-bot: [operations/mediawiki-config@master] db-eqiad,db-codfw.php: Repool db1101:3318, db2085:3318, db2083 https://gerrit.wikimedia.org/r/466710TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 466710 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad,db-codfw.php: Repool db1101:3318, db2085:3318, db2083 https://gerrit.wikimedia.org/r/466710TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-11T15:12:50Z] Stop MySQL on db2085:3318 to reclone db1101:3318 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 466658 merged by jenkins-bot: [operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318 https://gerrit.wikimedia.org/r/466658TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 466658 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318 https://gerrit.wikimedia.org/r/466658TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 466652 merged by jenkins-bot: [operations/mediawiki-config@master] db-eqiad.php: Repool db1099:3318 https://gerrit.wikimedia.org/r/466652TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 466652 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Repool db1099:3318 https://gerrit.wikimedia.org/r/466652TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. We believe the most important issues (missing revisions, pages and users, and making accessible the content) has been fixed. We will lower the UBN after doing some additional sanity checks. Having said that, for the following days, some missing things like user preferences, what links here* pages and others may act strange. A longer term fix (reimport from a good state) will slowly be deployed in the following days.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-11T14:30:44Z] Synchronized wmf-config/db-eqiad.php: T206743: mariadb: Depool db1087 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-11T14:28:17Z] depooling db1087 (T206743)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. applied a fix to: db1109 db1071 db1104 db1101:3318 db1092 dbstore1002TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-11T13:23:44Z] Stop MySQL on db2083 to reclone db1116:3318 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-11T13:20:24Z] Stop MySQL on db1116:3318 to reclone it from db2083 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 466594 merged by jenkins-bot: [operations/mediawiki-config@master] db-codfw.php: Depool db2083 https://gerrit.wikimedia.org/r/466594TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Marostegui added a comment. s8 host cloning process labsdb1011 labsdb1010 labsdb1009 dbstore1002 (not sure we should reclone it, it will be replaced "soon") db1124 (sanitarium) db1116 (backup source) db1109 db1104 (candidate master) db1101 (recentchanges) db1099 (recentchanges) db1092 db1087 (vslow and sanitarium master, probably better be done with mydumper+myloader) db1071 (master) TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
gerritbot added a comment. Change 466594 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-codfw.php: Depool db2083 https://gerrit.wikimedia.org/r/466594TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, Nandana, Banyek, jijiki, Mringgaard, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Volans, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
jcrespo added a comment. We know the exact timestamps of missing rows, from db1071-bin.007238:795791989 2018-09-13 09:08:17 on the active (codfw) master: db2045-bin.005879:1036765620 to db1071-bin.007238:796727644 2018-09-13 09:58:26 on the active (codfw) master: db2045-bin.005880:132937482 We have recovered all sql code executed at that time, no there should not be permanent data loss, I am trying to recover those now in a way that will be safe do no further loss happens of newer transactions- this may take a bit, as this happened a long time ago. If this had been identified 3 days later, we could not even do that, as binlog is purged after 30 days. In the worse case scenario, we could publish those changes so they are redone, but I am going to try to apply them to the main affected tables (user, revision, page, text, ...) automatically. Content itself was not affected because content is stored through a different replication channel- so only the metadata needed to access it.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, Nandana, Banyek, jijiki, Mringgaard, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Volans, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-11T11:10:24Z] Stop MYSQL on db2085:3318 and db1099:3318 T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, Nandana, Banyek, jijiki, Mringgaard, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Volans, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-11T11:09:49Z] Stop MYSQL on db2088:3318 and db1099:3318 T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, Nandana, Banyek, jijiki, Mringgaard, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Volans, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs