[Wikidata-bugs] [Maniphest] T272571: Updating's Wikidata property suggester caused replica lag on all wikidata databases

[hoo]

hoo closed this task as "Resolved".
hoo claimed this task.
hoo added a comment.


  I just did another suggester update, with the script described above 
(T272571#6771954 ) and we 
encountered no issues this time, thus this should be fine.

TASK DETAIL
  https://phabricator.wikimedia.org/T272571

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo
Cc: Lucas_Werkmeister_WMDE, Ladsgroup, Addshore, Marostegui, hoo, Aklapper, 
jcrespo, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Verdy_p, Wikidata-bugs, 
aude, Sjoerddebruin, Jdforrester-WMF, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T272571: Updating's Wikidata property suggester caused replica lag on all wikidata databases

[Addshore]

Addshore added a comment.


  We should be tackling this topic as part of T72037: [Story] Automate Entity 
Suggester Data Updates  which will 
likely lead to much refactoring and usage of a different update mechanism.

TASK DETAIL
  https://phabricator.wikimedia.org/T272571

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Addshore, Marostegui, hoo, Aklapper, jcrespo, Akuckartz, darthmon_wmde, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Verdy_p, Wikidata-bugs, aude, Sjoerddebruin, 
Jdforrester-WMF, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T272571: Updating's Wikidata property suggester caused replica lag on all wikidata databases

[hoo]

hoo added a comment.
Restricted Application added a project: wdwb-tech-focus.


  Currently the procedure for updating the table involves a hand-written 
script, which does data clean up (deletes LOTS of rows) in the database table 
after the dataset is inserted (which is probably what was causing these 
issues). See PropertySuggester_update 
.
  
  I spent a bit of time today on this and re-wrote this as a small python 
 script which can be 
applied on the dataset before it is inserted into the table (thus no more 
deletes), effectively solving the issues at hand. For the next update, I will 
use that and we can closely monitor the logs in order to conclusively close 
this.

TASK DETAIL
  https://phabricator.wikimedia.org/T272571

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo
Cc: Marostegui, hoo, Aklapper, jcrespo, Akuckartz, darthmon_wmde, Nandana, 
Lahi, Gq86, Pablo-WMDE, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Verdy_p, Wikidata-bugs, aude, Sjoerddebruin, 
Jdforrester-WMF, Addshore, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T272571: Updating's Wikidata property suggester caused replica lag on all wikidata databases

[jcrespo]

jcrespo created this task.
jcrespo added projects: Wikidata, MediaWiki-extensions-PropertySuggester, 
Wikimedia-production-error.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  We got an alert on `#wikimedia-databases` IRC saying:
  
PROBLEM - MariaDB sustained replica lag on db is CRITICAL: 2 ge 2 
https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag 
https://grafana.wikimedia.org/d/00273/mysql?orgId=1=db=9104
  
  However, the spike of lag was seen on all servers,it  just happened db 
was more sensitive to be reported:
  
  F34010823: Screenshot from 2021-01-21 10-57-14.png 

  
  Logstash indicated thousands of client errors due to lag:
  
  F34010825: Screenshot from 2021-01-21 10-59-06.png 

  
  Log around that time seems to indicate maintenance to Wikidata property 
suggester:
  
09:44:09  !log Updated the Wikidata property suggester with data from 
the 2021-01-11 JSON dump and applied the T132839 workarounds
  
  hoo on IRC seemed to agree that it was likely the cause:
  
09:48:10  The maintenance script rebuilds the entire table (yuck...)
  
  We can have temporary lag on one server and we are able to cope with that, 
but on all servers is quite imapacting for editors/recentchanges/etc.
  
  Let's try to avoid production impact by one of: refactoring the script, 
adding pauses (e.g. waitForReplica()) or avoiding its run, or any other method, 
up to devel team.

TASK DETAIL
  https://phabricator.wikimedia.org/T272571

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: Marostegui, hoo, Aklapper, jcrespo, Akuckartz, darthmon_wmde, Nandana, 
Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Verdy_p, Wikidata-bugs, aude, Sjoerddebruin, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs