subject:"\[Wikidata\-bugs\] \[Maniphest\] \[Commented On\] T206743\: S8 replication issues leading to rows missing during eqiad \-> codfw switch \(Was\: \"A few lexemes disappeared\"\)"

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-12-12 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-12-12T08:37:00Z]  Remove old backup directory from db1116 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, StashbotCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, _jensen, D3r1ck01, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-12-03 Thread Marostegui

Marostegui added a comment.
If there is no space issues on that host, there is no harm in leaving it a bit longer I would say.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, _jensen, D3r1ck01, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-12-03 Thread Banyek

Banyek added a comment.
on db1116 we still have the backup dir:

drwxr-xr-x  8 mysql mysql 8.0K Oct 11 13:21 sqldata.s8_BACKUP_T206743

I guess we shall it remove now, as this ticket is resolved (and we still remember the issue) ?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, BanyekCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, _jensen, D3r1ck01, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-25 Thread jcrespo

jcrespo added a comment.
@Pigsonthewing I hope my comment at Wikidata Village Pump was helpful- if you think that is ok, I would suggest closing this task, and open a different one to track the merges of old history (this was to track the recovery from backups)?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-24 Thread Addshore

Addshore added a comment.

In T206743#4691328, @jcrespo wrote:
@Addshore I thought you had communicated to wikidata users about that? Apparently not, or @Pigsonthewing didn't see it, could you link your messages to him?


I'll leave that to @Lea_Lacroix_WMDE. This is still on the needs announcement column for us.
The related pages that I created can be found @ https://www.wikidata.org/wiki/User:Addshore/2018/10/DC_Switch_IssueTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, AddshoreCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-24 Thread jcrespo

jcrespo added a comment.
@Addshore I thought you had communicated to wikidata users about that? Apparently not, or @Pigsonthewing didn't see it, could you link your messages to him?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-24 Thread Marostegui

Marostegui added a comment.
I guess it is because what @Addshore described at T206743#4662054 ?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-24 Thread Marostegui

Marostegui added a comment.
Final round of tables that have been checked and are clean:

abuse_filter_action
abuse_filter_history
betafeatures_user_counts
content
content_models
global_block_whitelist
ipblocks
math
mathoid
ores_model
protected_titles
querycache_info
revision_comment_temp
securepoll_elections
securepoll_entity
securepoll_lists
securepoll_msgs
securepoll_voters
securepoll_properties
sites
site_identifiers
site_stats
slot_roles
templatelinks
translate_metadata
user_former_groups
user_groups

At this point, all the tables in s8 have been checked and confirmed clean.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-23 Thread Marostegui

Marostegui added a comment.
The following tables are empty so no need to check:

actor
config
filearchive
filejournal
globalblocks
hidden
image
image_comment_temp
interwiki
ipblocks_restrictions
job
l10n_cache
oldimage
revision_actor_temp
searchindex
securepoll_cookie_match
securepoll_options
securepoll_questions
securepoll_strike
securepoll_votes
transcache
transcode
updatelog
uploadstashTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-23 Thread Marostegui

Marostegui added a comment.
wb_terms is finished and it is all clean.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-23 Thread Marostegui

Marostegui added a comment.
More tables to check after the ones checked at: T206743#4685983
abuse_filter_log (T206743#4685983) was fixed by Jaime (I have rechecked again and it is indeed now consistent).

The following tables are clean between db1087 and db1092:

abuse_filter
category
change_tag_def
cu_changes
externallinks
geo_tags
imagelinks
iwlinks
linter
log_search
ores_classification
page
page_props
page_restrictions
pagelinks
spoofuser
tag_summary
templatelinks
user
user_newtalk
user_properties
watchlist
wb_changes_subscription
wb_items_per_site
wbc_entity_usage

The following tables reported differences and have already been fixed by Jaime (and confirmed after a second check by me)

change_tag

On-going check for the last table (which takes several hours:

wb_termsTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-23 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-23T06:39:09Z]  Stop replication on db1092 and db1087 for checking T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-22 Thread Marostegui

Marostegui added a comment.
These tables are fine:

logging
revision
slots
text

Starting to check:

abuse_filter
category 
change_tag 
change_tag_def 
cu_changes 
externallinks 
geo_tags
imagelinks
ip_changes 
iwlinks 
linter 
log_search 
ores_classification
page
page_props 
page_restrictions
pagelinks 
spoofuser 
tag_summary 
templatelinks
user
user_newtalk
user_properties
watchlist
wb_changes_subscription 
wb_items_per_site
wbc_entity_usageTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-22 Thread Addshore

Addshore added a comment.

In T206743#4685334, @jcrespo wrote:
@Addshore did you sent to wikidata users the list you compiled to check bot activity?


YesTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-22 Thread Marostegui

Marostegui added a comment.
The following tables have been re-checked on db1092 (recloned from codfw) vs db1087 (manually fixed):

abuse_filter_log
archive
babel
bv2013_edits
bv2015_edits
bv2017_edits
categorylinks
comment
cu_log
ipblocks
langlinks
module_deps
object_cache
querycache
querycachetwo
recentchanges
redirect
revtag
translate_reviews
translate_groupreviews
translate_groupstats
translate_messageindex
translate_sections
valid_tag
wb_changes
wb_changes_dispatch
wb_id_counters
wb_property_info
wbqc_constraints
wbs_propertypairs

On-going:

logging
revision
slots
text

Found differences on:
abuse_filter_log which has 66 rows more on db1092 than on db1087 (and sanitarium, and labs) and db1071 (master).TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-22 Thread jcrespo

jcrespo added a comment.
After many fixes during the weekend, wb_terms also fixed on labs, all hosts should be consistent now, doing some extra checks to verify everything is good.

@Addshore did you sent to wikidata users the list you compiled to check bot activity?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-19 Thread Marostegui

Marostegui added a comment.
Update: pagelinks is now being re-imported on labs (this table is around 136G) so it will take a whileTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-18 Thread Marostegui

Marostegui added a comment.
Update from Jaime 18th Oct 16:05: s8 core hosts all finished getting fixed (pending labs)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-18 Thread Marostegui

Marostegui added a comment.

In T206743#4676466, @ArielGlenn wrote:

In T206743#4675755, @Banyek wrote:
on db1124 with instance s8 we have a repliation error as

Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db1087-bin.003073, end_log_pos 582738698


Where was this error message logged? I have been looking for an error log on db1124 without success.


Probably on show slave status, but that error was fixed already.
This host is sanitarium, so it is not user facing error (it does create lag on labs though).

This host is expected to fail while it is getting fixed.
Right now there is another error there, as replication got broken again.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-18 Thread ArielGlenn

ArielGlenn added a comment.

In T206743#4675755, @Banyek wrote:
on db1124 with instance s8 we have a repliation error as

Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db1087-bin.003073, end_log_pos 582738698


Where was this error message logged? I have been looking for an error log on db1124 without success.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, ArielGlennCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-17 Thread Marostegui

Marostegui added a comment.
Update from 17th at 19:04 from Jaime:
all tables except wb_terms, which is half done, should be equal on the s8 masterTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-17 Thread Banyek

Banyek added a comment.
on db1124 with instance s8 we have a repliation error as

Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db1087-bin.003073, end_log_pos 582738698TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, BanyekCc: Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-17 Thread Marostegui

Marostegui added a comment.
Update from Jaime at 12:10 UTC
All tables on s8 master fixed except pagelinks and wb_termsTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-17 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-17T06:52:01Z]  fixing s8 master drifts T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, ArielGlenn, Lydia_Pintscher, Darkdadaah, He7d3r, TheDJ, Jdforrester-WMF, Mbch331, Jay8g, Krenair, akosiaris, greg, Aklapper___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-17 Thread Marostegui

Marostegui added a comment.
Update from yesterday at around 14:00UTC
@jcrespo has done an amazing job of manually checking and fixing most of the tables on db1087 (which is the labs master, and it is not as easy to reclone as the others). He's gone thru all the tables to check and fix them.
Right all the tables apart from pagelinks and wb_terms are supposed to be fixed  (although replication is broken on db1124 sanitarium master for wb_items_per_site, which is kind of expected with such amount of rows to check - we are looking at it).

pagelinks and wb_terms are huge tables, and they cannot be diffed at once because our mysql client reaches OOM, so these checks need to be split on smaller diffs, that's why it is taking a bit longer.

Thanks Jaime for this huge work.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-16 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-16T06:05:03Z]  stopping db1092 and db1087 in sync T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread Addshore

Addshore added a comment.
I have prepared https://www.wikidata.org/wiki/User:Addshore/2018/10/DC_Switch_Issue which includes the pages that will need to be reviewed.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread Marostegui

Marostegui added a comment.
So as Jaime said at T206743#4666169 all the pooled replicas in eqiad have now the content from codfw, so that is now consistent. 
What is pending is db1087 (depooled) which is the master for db1124 (labsdb master) and labsdb, and of course the master, db1071 which doesn't receive reads.
So labsdb still have inconsistencies.

We are now working on fixing those, db1087 which will automatically (via replication) will fix db1124 and labs.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T13:16:48Z]  stopping db1092 and db1087 in sync T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread jcrespo

jcrespo added a comment.
I'll generate a list once all slaves that serve mw traffic are re imaged :)

This is true right now for all pooled read only replicas, but I need to fix the master.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread Addshore

Addshore added a comment.

In T206743#4665661, @jcrespo wrote:
@Addshore The main task now is to check pages and pages from revisions that were lost and then restored, and check edits done since last wednesday on those pages, as, while the data has not been lost, those may have put an  edit over an older version as the latest, hiding some data. We need to generate a list of Qs to review.


I'll generate a list once all slaves that serve mw traffic are re imaged :)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T09:46:16Z]  Synchronized wmf-config/db-eqiad.php: Depool db1092 for recloning - T206743 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread jcrespo

jcrespo added a comment.
@Addshore The main task now is to check pages and pages from revisions that were lost and then restored, and check edits done since last wednesday on those pages, as, while the data has not been lost, those may have put an  edit over an older version as the latest, hiding some data. We need to generate a list of Qs to review.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T08:46:38Z]  Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 - T206743 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread gerritbot

gerritbot added a comment.
Change 467285 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1104

https://gerrit.wikimedia.org/r/467285TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread Addshore

Addshore added a comment.
I wrote a little crappy but effective query that I can just run on all of the shards to see if anything else was affected.

select max(rev_id), min(rev_id), count(*), max(rev_id) - min(rev_id) as diff from ( select rev_id, rev_timestamp from revision order by rev_id DESC limit 2000 ) as something where rev_timestamp > 20180913090816 and rev_timestamp < 20180913095825;

It shows us what happened to the rev_id in the 50 minute period that we are concerned with.
Small difference could be put down to revdel etc, as long as we don't see anything that is near 0 everything should be fine.

S1 - enwiki

+-+-+--+--+
| max(rev_id) | min(rev_id) | count(*) | diff |
+-+-+--+--+
|   859332479 |   859327787 | 4604 | 4692 |
+-+-+--+--+

S2 - ptwiki

+-+-+--+--+
| max(rev_id) | min(rev_id) | count(*) | diff |
+-+-+--+--+
|53121516 |53121457 |   58 |   59 |
+-+-+--+--+

S3 - afwiki

+-+-+--+--+
| max(rev_id) | min(rev_id) | count(*) | diff |
+-+-+--+--+
| 1814203 | 1814188 |   16 |   15 |
+-+-+--+--+

S4 - commonswiki

+-+-+--+--+
| max(rev_id) | min(rev_id) | count(*) | diff |
+-+-+--+--+
|   319952253 |   319947702 | 4387 | 4551 |
+-+-+--+--+

S5 - dewiki

+-+-+--+--+
| max(rev_id) | min(rev_id) | count(*) | diff |
+-+-+--+--+
|   180869513 |   180868285 | 1188 | 1228 |
+-+-+--+--+

S6 - frwiki

+-+-+--+--+
| max(rev_id) | min(rev_id) | count(*) | diff |
+-+-+--+--+
|   152158909 |   152157584 | 1301 | 1325 |
+-+-+--+--+

S7 - arwiki

+-+-+--+--+
| max(rev_id) | min(rev_id) | count(*) | diff |
+-+-+--+--+
|30515544 |30515320 |  220 |  224 |
+-+-+--+--+

So it looks like s8 was indeed the only issue @MarosteguiTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread jcrespo

jcrespo added a comment.
@Pigsonthewing Please don't get worry- as I said at T206743#4658620 only primary data was recovered, cache tables may show still wrong data until we fully recover everything this week, showing some weird data. Please be patient, but don't touch anything that seems wrong, as it will only complicate the full recovery we are doing at the moment.

Once the recovery is fully done, we will announce it here and you should have everything back- however if you edit or try to recover manually the edits, it will generate new revisions and that version will stay like that- as it will be the latest one (older versions will be stored in the history- and it may show as if you vandalized (removed content) from an article.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread gerritbot

gerritbot added a comment.
Change 467285 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1104

https://gerrit.wikimedia.org/r/467285TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T07:24:08Z]  Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 48s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T07:12:16Z]  Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread gerritbot

gerritbot added a comment.
Change 467261 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/467261TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread gerritbot

gerritbot added a comment.
Change 467261 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/467261TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread gerritbot

gerritbot added a comment.
Change 467260 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Fully repool db1109

https://gerrit.wikimedia.org/r/467260TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread gerritbot

gerritbot added a comment.
Change 467260 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Fully repool db1109

https://gerrit.wikimedia.org/r/467260TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread gerritbot

gerritbot added a comment.
Change 467252 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1109

https://gerrit.wikimedia.org/r/467252TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-15 Thread gerritbot

gerritbot added a comment.
Change 467252 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1109

https://gerrit.wikimedia.org/r/467252TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-14 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T05:16:12Z]  Stop MySQL on db1109 for recloning - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-14 Thread gerritbot

gerritbot added a comment.
Change 467249 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/467249TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-14 Thread gerritbot

gerritbot added a comment.
Change 467249 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/467249TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-13 Thread Pigsonthewing

Pigsonthewing added a comment.
The item I gave as an example above still shows the wrong image:

https://www.wikidata.org/wiki/Q2058295

It is also missing other data. Note this diff, the most recent edit (as I write):

https://www.wikidata.org/w/index.php?title=Q2058295=762891812=745457545

as it was made with Mix'n'Match, only one ID should have  been added.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, PigsonthewingCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-13 Thread Addshore

Addshore added a comment.
Here is a list of all pages that will likely be affected already with data being lost in the revisions that have since been restored:

MariaDB [wikidatawiki]>  SELECT DISTINCT rev_page, page_title, page_namespace, page_touched
-> FROM revision AS revision
-> INNER JOIN page AS page ON revision.rev_page = page.page_id
-> WHERE rev_id > 745452473 and rev_id < 745468910 and page_touched > 20181010151500
-> ORDER BY page_namespace, page_title;
+--+-+++
| rev_page | page_title  | page_namespace | page_touched   |
+--+-+++
|   957791 | Q1008351|  0 | 20181011183506 |
|   972072 | Q1022718|  0 | 2018102403 |
| 11573912 | Q10269330   |  0 | 20181011102631 |
| 11618130 | Q10313336   |  0 | 20181011183306 |
| 12075277 | Q10801324   |  0 | 20181011231428 |
|  1037514 | Q1089690|  0 | 20181011224900 |
|   115186 | Q112562 |  0 | 20181011232641 |
|  1072735 | Q1125675|  0 | 2018102401 |
|  1088182 | Q1141499|  0 | 20181011093245 |
|  1098210 | Q1151801|  0 | 2018102357 |
|  1103257 | Q1156971|  0 | 20181011231351 |
|  1140371 | Q1196895|  0 | 20181011150707 |
|   130504 | Q128504 |  0 | 20181012061613 |
|  1246692 | Q1306726|  0 | 20181012061648 |
|  1278081 | Q1338401|  0 | 20181011010159 |
|  1304828 | Q1365309|  0 | 2018102351 |
|   140372 | Q138671 |  0 | 20181012114338 |
|   140517 | Q138820 |  0 | 20181011032316 |
|  1373695 | Q1435223|  0 | 20181011093244 |
|  1444721 | Q1507379|  0 | 2018102349 |
| 17026621 | Q15378860   |  0 | 20181011195823 |
|  1480008 | Q1543446|  0 | 2018102346 |
| 17087585 | Q15443105   |  0 | 2018102023 |
| 17343834 | Q15708744   |  0 | 20181012023419 |
| 17825650 | Q16199113   |  0 | 20181010184808 |
| 17865127 | Q16240975   |  0 | 20181012124819 |
|  1575620 | Q1641483|  0 | 2018102343 |
|  1607340 | Q1673849|  0 | 20181011200423 |
|   168071 | Q167548 |  0 | 2018102132 |
|  1611743 | Q1678281|  0 | 2018102340 |
|  1616952 | Q1683527|  0 | 20181011102520 |
|  1624134 | Q1690779|  0 | 2018102338

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Marostegui

Marostegui added a comment.
root@neodymium:/home/marostegui# ./section s8 | while read host port; do echo $host; mysql.py -h$host:$port wikidatawiki -BN -e "select page_id, page_title, page_latest from page where page_id = 99480;";done
labsdb1011.eqiad.wmnet
99480	Q97215	745463455
labsdb1010.eqiad.wmnet
99480	Q97215	745463455
labsdb1009.eqiad.wmnet
99480	Q97215	745463455
dbstore2001.codfw.wmnet
99480	Q97215	745463455
dbstore1002.eqiad.wmnet
99480	Q97215	745463455
db2094.codfw.wmnet
99480	Q97215	745463455
db2086.codfw.wmnet
99480	Q97215	745463455
db2085.codfw.wmnet
99480	Q97215	745463455
db2083.codfw.wmnet
99480	Q97215	745463455
db2082.codfw.wmnet
99480	Q97215	745463455
db2081.codfw.wmnet
99480	Q97215	745463455
db2080.codfw.wmnet
99480	Q97215	745463455
db2079.codfw.wmnet
99480	Q97215	745463455
db2045.codfw.wmnet
99480	Q97215	745463455
db1124.eqiad.wmnet
99480	Q97215	745463455
db1116.eqiad.wmnet
99480	Q97215	745463455
db1109.eqiad.wmnet
99480	Q97215	745463455
db1104.eqiad.wmnet
99480	Q97215	745463455
db1101.eqiad.wmnet
99480	Q97215	745463455
db1099.eqiad.wmnet
99480	Q97215	745463455
db1092.eqiad.wmnet
99480	Q97215	745463455
db1087.eqiad.wmnet
99480	Q97215	745463455
db1071.eqiad.wmnet
99480	Q97215	745463455TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Addshore

Addshore added a comment.
Using the example that I had in T206743#4661943 the pages that were fixable should now be fixed

MariaDB [wikidatawiki]> select page_id, page_title, page_latest from page where page_id = 99480;
+-++-+
| page_id | page_title | page_latest |
+-++-+
|   99480 | Q97215 |   745463455 |
+-++-+
1 row in set (0.00 sec)

MariaDB [wikidatawiki]> select rev_id from revision where rev_page = 99480 order by rev_id desc limit 2;
+---+
| rev_id|
+---+
| 745463455 |
| 737776142 |
+---+
2 rows in set (0.00 sec)

@Marostegui I guess you could confirm that using the powers you demonstrated in T206743#4662022 ? :)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-12T18:37:06Z]  modified attachLatest.php script finished running over 9395 pages T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Addshore

Addshore added a comment.

In T206743#4662033, @Marostegui wrote:

In T206743#4662023, @Addshore wrote:
I'll run a maint script to fix the page_latest of the 9000ish pages that have an incorrect one so that we don't end up with more inconsistencies / broken stuff moving forward.


But what will happen with the correct data we have in codfw??


Well after the script run both DCs should be correct.

Essentially what has happened is:


Switch from codfw to eqiad, resulting in the missing transactions
One of the pages that had missing updates was edited during that time, making a new revision (with a newer rev id) and bumping the page_latest
When we then added the transactions back, the previous updates to the page table changes the page_latest back to an ID prior to what is currently the latest.


Once the script has finished the fallout of this whole thing should be that there are some inconsistencies with the pages / history for the pages listed in T206743#4661906 which were edited during this odd time between the switch back to eqiad and the adding of these revisions / running of this script.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Marostegui

Marostegui added a comment.

In T206743#4662023, @Addshore wrote:
I'll run a maint script to fix the page_latest of the 9000ish pages that have an incorrect one so that we don't end up with more inconsistencies / broken stuff moving forward.


But what will happen with the correct data we have in codfw??TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-12T18:25:26Z]  running modified attachLatest.php script over ~9000 pages on wikidatawiki (with added wait for slaves) T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Addshore

Addshore added a comment.
I'll run a maint script to fix the page_latest of the 9000ish pages that have an incorrect one so that we don't end up with more inconsistencies / broken stuff moving forward.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Marostegui

Marostegui added a comment.

In T206743#4661943, @Addshore wrote:
Which means there are ~9433 pages that now probably have the wrong page_latest, for example:


We are fixing that by recloning the hosts (T206743#4658362) from codfw, which have the correct data:
See how db1099 and db1101 already have the good data.

root@neodymium:/home/marostegui# ./section s8 | while read host port; do echo $host; mysql.py -h$host:$port wikidatawiki -BN -e "select page_id, page_title, page_latest from page where page_id = 99480;";done
labsdb1011.eqiad.wmnet
99480	Q97215	737776142
labsdb1010.eqiad.wmnet
99480	Q97215	737776142
labsdb1009.eqiad.wmnet
99480	Q97215	737776142
dbstore2001.codfw.wmnet
99480	Q97215	745463455
dbstore1002.eqiad.wmnet
99480	Q97215	737776142
db2094.codfw.wmnet
99480	Q97215	745463455
db2086.codfw.wmnet
99480	Q97215	745463455
db2085.codfw.wmnet
99480	Q97215	745463455
db2083.codfw.wmnet
99480	Q97215	745463455
db2082.codfw.wmnet
99480	Q97215	745463455
db2081.codfw.wmnet
99480	Q97215	745463455
db2080.codfw.wmnet
99480	Q97215	745463455
db2079.codfw.wmnet
99480	Q97215	745463455
db2045.codfw.wmnet
99480	Q97215	745463455
db1124.eqiad.wmnet
99480	Q97215	737776142
db1116.eqiad.wmnet
99480	Q97215	745463455
db1109.eqiad.wmnet
99480	Q97215	737776142
db1104.eqiad.wmnet
99480	Q97215	737776142
db1101.eqiad.wmnet
99480	Q97215	745463455
db1099.eqiad.wmnet
99480	Q97215	745463455
db1092.eqiad.wmnet
99480	Q97215	737776142
db1087.eqiad.wmnet
99480	Q97215	737776142
db1071.eqiad.wmnet
99480	Q97215	737776142

The idea with the quick was to get some rows in place, rather than have no rows.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Marostegui

Marostegui added a comment.
We are also fully restoring the eqiad hosts from codfw which has all the fine data.
The fix done on Friday was a quick fix to get all the data in there.

On eqiad, db1099 and db1101 should already have the proper data. They were recloned from codfwTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Addshore

Addshore added a comment.
It looks like we could fix these with the attachLatest.php maint script but it would need a little bit of modification.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Addshore

Addshore added a comment.
Which means there are ~9433 pages that now probably have the wrong page_latest, for example:

MariaDB [wikidatawiki]> select page_id, page_title, page_latest from page where page_id = 99480;
+-++-+
| page_id | page_title | page_latest |
+-++-+
|   99480 | Q97215 |   737776142 |
+-++-+
1 row in set (0.00 sec)

MariaDB [wikidatawiki]> select rev_id from revision where rev_page = 99480 order by rev_id desc limit 2;
+---+
| rev_id|
+---+
| 745463455 |
| 737776142 |
+---+
2 rows in set (0.00 sec)

These pages can be found with the following query:

SELECT DISTINCT rev_page, page_title, page_namespace, page_touched
FROM revision AS revision
INNER JOIN page AS page ON revision.rev_page = page.page_id
WHERE rev_id > 745452473 and rev_id < 745468910 and page_touched < 20181010151500
ORDER BY page_namespace, page_title;TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Addshore

Addshore added a comment.

In T206743#4661704, @Pigsonthewing wrote:
I'm pretty sure I added an image:

https://commons.wikimedia.org/wiki/File:Sandy_Wollaston_(cropped).jpg

to:

https://www.wikidata.org/wiki/Q2058295

that day (though of course its possible I was distracted and didn't complete the task); it's not there now.


I see it in the revision history.

https://www.wikidata.org/w/index.php?diff=745457425=744814394=Q2058295

But the current parsed page does indeed show the wrong image.

F26560208: image.png

It looks like an edit was made and the wrong latest revision was chosen from the page table.
This means the edit was patched onto the version of the entity before the edits that were made during the 50 minute slot that was lost.

F26560328: image.png

It looks like we should write something to grab a list of pages that were affected and try to fix this.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Pigsonthewing

Pigsonthewing added a comment.
I'm pretty sure I added an image:

https://commons.wikimedia.org/wiki/File:Sandy_Wollaston_(cropped).jpg

to:

https://www.wikidata.org/wiki/Q2058295

that day (though of course its possible I was distracted and didn't complete the task); it's not there now.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, PigsonthewingCc: Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Addshore

Addshore added a comment.

In T206743#4661592, @Lea_Lacroix_WMDE wrote:
User:Jason.nlw told me that during the effected period, he was in the middle of a batch upload, and some data was not restored yet.  
 https://www.wikidata.org/w/index.php?title=Special:Contributions=20180913095817=500=user=Jason.nlw===2018-09-13=2018-09-13

Can someone have a look? Thanks.


It looks like all of the revisions for that user exist in both data centers:

EQIAD:

MariaDB [wikidatawiki]> select count(*) from revision where rev_user = ( SELECT user_id from user where user_name = 'Jason.nlw' );
+--+
| count(*) |
+--+
|   244293 |
+--+
1 row in set (0.14 sec)

CODFW:

MariaDB [wikidatawiki]> select count(*) from revision where rev_user = ( SELECT user_id from user where user_name = 'Jason.nlw' );
+--+
| count(*) |
+--+
|   244293 |
+--+
1 row in set (0.21 sec)

Do you have any other details?
It could be that the user was just doing the batch upload during the data centre switch over period, at which time wikidata was in read only mode for a few minutes.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-12 Thread Lea_Lacroix_WMDE

Lea_Lacroix_WMDE added a comment.
User:Jason.nlw told me that during the effected period, he was in the middle of a batch upload, and some data was not restored yet.  
https://www.wikidata.org/w/index.php?title=Special:Contributions=20180913095817=500=user=Jason.nlw===2018-09-13=2018-09-13

Can someone have a look? Thanks.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, Lea_Lacroix_WMDECc: Nikerabbit, gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread gerritbot

gerritbot added a comment.
Change 466710 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Repool db1101:3318, db2085:3318, db2083

https://gerrit.wikimedia.org/r/466710TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread gerritbot

gerritbot added a comment.
Change 466710 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Repool db1101:3318, db2085:3318, db2083

https://gerrit.wikimedia.org/r/466710TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Johan, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, TheDJ, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T15:12:50Z]  Stop MySQL on db2085:3318 to reclone db1101:3318 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread gerritbot

gerritbot added a comment.
Change 466658 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/466658TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread gerritbot

gerritbot added a comment.
Change 466658 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/466658TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread gerritbot

gerritbot added a comment.
Change 466652 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1099:3318

https://gerrit.wikimedia.org/r/466652TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread gerritbot

gerritbot added a comment.
Change 466652 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1099:3318

https://gerrit.wikimedia.org/r/466652TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread jcrespo

jcrespo added a comment.
We believe the most important issues (missing revisions, pages and users, and making accessible the content) has been fixed. We will lower the UBN after doing some additional sanity checks. Having said that, for the following days, some missing things like user preferences, what links here* pages and others may act strange. A longer term fix (reimport from a good state) will slowly be deployed in the following days.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T14:30:44Z]  Synchronized wmf-config/db-eqiad.php: T206743: mariadb: Depool db1087 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T14:28:17Z]  depooling db1087 (T206743)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread jcrespo

jcrespo added a comment.
applied a fix to:

db1109
db1071
db1104
db1101:3318
db1092
dbstore1002TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T13:23:44Z]  Stop MySQL on db2083 to reclone db1116:3318 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T13:20:24Z]  Stop MySQL on db1116:3318 to reclone it from db2083 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread gerritbot

gerritbot added a comment.
Change 466594 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2083

https://gerrit.wikimedia.org/r/466594TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread Marostegui

Marostegui added a comment.
s8 host cloning process


 labsdb1011
 labsdb1010
 labsdb1009
 dbstore1002 (not sure we should reclone it, it will be replaced "soon")
 db1124 (sanitarium)
 db1116 (backup source)
 db1109
 db1104 (candidate master)
 db1101 (recentchanges)
 db1099 (recentchanges)
 db1092
 db1087 (vslow and sanitarium master, probably better be done with mydumper+myloader)
 db1071 (master)
TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, CucyNoiD, Nandana, NebulousIris, Banyek, jijiki, Mringgaard, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Volans, Maathavan, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread gerritbot

gerritbot added a comment.
Change 466594 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2083

https://gerrit.wikimedia.org/r/466594TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: gerritbot, WMDE-leszek, jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, Nandana, Banyek, jijiki, Mringgaard, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Volans, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread jcrespo

jcrespo added a comment.
We know the exact timestamps of missing rows,
from

db1071-bin.007238:795791989 2018-09-13 09:08:17 on the active (codfw) master: db2045-bin.005879:1036765620

to

db1071-bin.007238:796727644 2018-09-13 09:58:26 on the active (codfw) master: db2045-bin.005880:132937482

We have recovered all sql code executed at that time, no there should not be permanent data loss, I am trying to recover those now in a way that will be safe do no further loss happens of newer transactions- this may take a bit, as this happened a long time ago. If this had been identified 3 days later, we could not even do that, as binlog is purged after 30 days.

In the worse case scenario, we could publish those changes so they are redone, but I am going to try to apply them to the main affected tables (user, revision, page, text, ...) automatically. Content itself was not affected because content is stored through a different replication channel- so only the metadata needed to access it.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: jcrespo, mark, Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, Nandana, Banyek, jijiki, Mringgaard, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Volans, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T11:10:24Z]  Stop MYSQL on db2085:3318 and db1099:3318 T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, Nandana, Banyek, jijiki, Mringgaard, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Volans, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206743: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared")

2018-10-11 Thread Stashbot

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T11:09:49Z]  Stop MYSQL on db2088:3318 and db1099:3318 T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Stashbot, Nikki, Marostegui, daniel, TerraCodes, Liuxinyu970226, Addshore, Ladsgroup, Lea_Lacroix_WMDE, Lexicographical data, KaMan, Nandana, Banyek, jijiki, Mringgaard, AndyTan, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Volans, Jonas, Luke081515, Wikidata-bugs, aude, Lydia_Pintscher, Darkdadaah, Mbch331, Jay8g, Krenair, akosiaris___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

85 matches

Mail list logo