[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Addshore added a comment. 5:59 PM !log killed rebuildItemTerms on mwmaint1002 5:59 PM <+stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL PREFERENCES https://phabricator.wikimedia.org/settin

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Addshore added a comment. Looks like API request rate is recovering F30928576: image.png TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addsh

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

gerritbot added a comment. Change 547264 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [operations/puppet@production] mediawiki: Temporary disable rebuildTermIndex https://gerrit.wikimedia.org/r/547264 TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

gerritbot added a comment. Change 547264 **merged** by Giuseppe Lavagetto: [operations/puppet@production] mediawiki: Temporary disable rebuildTermIndex https://gerrit.wikimedia.org/r/547264 TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL PREFERENCES https://phabricator.

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Addshore added a comment. Edit rate recovered, then got worse again F30928660: image.png We are seeing 5-15s lag on db slaves. F30928662: image.png Going to revert the 2 backports TA

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-10-30T17:26:36Z] Synchronized php-1.35.0-wmf.3/extensions/Wikibase: Revert 16:02 UTC T236928 (duration: 01m 04s) TASK DETAIL https://phabricator.wikimedia.org/T236928 EMA

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-10-30T17:29:15Z] Synchronized php-1.35.0-wmf.4/extensions/Wikibase: Revert 16:05 UTC T236928 (duration: 01m 05s) TASK DETAIL https://phabricator.wikimedia.org/T236928 EMA

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Addshore added a comment. The replag all seemed to come from one host which was having issues https://grafana.wikimedia.org/d/00273/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1126&var-port=9104 That seems to have recovered now. Edits seem to be recovering

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

jcrespo added a comment. > We are seeing 5-15s lag on db slaves. Offtopic- I don't believe the lag was real- it just that there was so much overload that lag checking reads/connections by mw would take so much time to finish and return results. This is irrelevant for this ticket (the sta

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

jcrespo added a comment. > The maintenance script restarted every 30 mins Small correction, every hour at XX:30 minutes. Maybe you already knew that, but just in case it could lead to missleading analysis. TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Addshore added a comment. In T236928#5620493 , @jcrespo wrote: >> The maintenance script restarted every 30 mins > > Small correction, every hour at XX:30 minutes. Maybe you already knew that, but just in case it could lead to missle

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Bugreporter added a comment. This may deserve an incident documentation . TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup,

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Addshore added a comment. Looking at things, everything still looks nice and recovered. In T236928#5621066 , @Bugreporter wrote: > This may deserve an incident documentation

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

gerritbot added a comment. Change 547397 had a related patch set uploaded (by Addshore; owner: Addshore): [mediawiki/extensions/Wikibase@master] Revert "Stop locking and use DISTINCT when finding used terms to delete" https://gerrit.wikimedia.org/r/547397 TASK DETAIL https://phabrica

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

gerritbot added a comment. Change 547397 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@master] Revert "Stop locking and use DISTINCT when finding used terms to delete" https://gerrit.wikimedia.org/r/547397 TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL PREFE

[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)

Addshore added a comment. Created a draft incident documentation https://wikitech.wikimedia.org/wiki/Incident_documentation/20191031-wikidata TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: La