[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Lucas_Werkmeister_WMDE added a comment. What @brennen reported isn’t a deadlock, isn’t it? It seems to be some race condition that we thought Should Never Happen™. TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. This is the right issue but given the size of migration still a little bit of deadlock is unavoidable. How big the number is? TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

brennen added a comment. I'm seeing a handful of these in 1.35.0-wmf.8: PHP Warning: [data-update-failed]: A data update callback triggered an exception (Fail-safe exception. Number of ids to be restored is not equal to the number of records that are about to be inserted into master.

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 547262 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@master] Add IGNORE on insert in ReplicaMasterAwareRecordIdsAcquirer https://gerrit.wikimedia.org/r/547262 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

jcrespo added a comment. Unless there is a last time change, this issue is for me fixed. Obviously, please do any data checks/refill that you think is necessary (if any) to fix potential problems caused by this. Thanks for working on this. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. Still pretty good: F31010064: image.png F31010066: image.png TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 548279 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.4] Retrying "Stop locking and use DISTINCT when finding used terms to delete" https://gerrit.wikimedia.org/r/548279 TASK DETAIL https://phabricator.wikimedia.org/T234948

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 547535 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@master] Retrying "Stop locking and use DISTINCT when finding used terms to delete" https://gerrit.wikimedia.org/r/547535 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 548279 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.4] Retrying "Stop locking and use DISTINCT when finding used terms to delete" https://gerrit.wikimedia.org/r/548279 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 547535 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [mediawiki/extensions/Wikibase@master] Revert "Revert "Stop locking and use DISTINCT when finding used terms to delete"" https://gerrit.wikimedia.org/r/547535 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. going back and forth and trying things in mysql in productions says if we stop using distinct and stop batching (using limit 1 instead) would make it wy faster. I make a patch for that. TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Lucas_Werkmeister_WMDE added a comment. It looks like `SELECT DISTINCT … WHERE … IN (…, …, …)` might not be the best query for “which of these IDs are still used?” when the IDs may be used millions of times. The “range” join type seems especially worrying, but even when it’s not used:

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. wikiadmin@10.64.32.198(wikidatawiki)> EXPLAIN SELECT /* Wikibase\Lib\Store\Sql\Terms\DatabaseItemTermStore::cleanTermsIfUnused */ DISTINCT wbit_term_in_lang_id AS `value` FROM `wbt_item_terms` WHERE wbit_term_in_lang_id IN ('828227', '828232', '828257',

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

jcrespo added a comment. FYI There seems to be a 50% or more write penalty throughput when the rebuild script is running: https://grafana.wikimedia.org/d/00273/mysql?var-server=db1109=9104=eqiad%20prometheus%2Fops=1=1572439082952=1572460682953 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 547262 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [mediawiki/extensions/Wikibase@master] Add IGNORE on insert in ReplicaMasterAwareRecordIdsAcquirer https://gerrit.wikimedia.org/r/547262 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-10-30T16:05:15Z] Synchronized php-1.35.0-wmf.4/extensions/Wikibase: Wikibase deadlock reduction, [[gerrit:547244|Stop locking and use DISTINCT when finding used terms to delete]] (T234948

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 547243 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.3] Stop locking and use DISTINCT when finding used terms to delete https://gerrit.wikimedia.org/r/547243 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 547244 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.4] Stop locking and use DISTINCT when finding used terms to delete https://gerrit.wikimedia.org/r/547244 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 547244 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.4] Stop locking and use DISTINCT when finding used terms to delete https://gerrit.wikimedia.org/r/547244 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 547243 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.3] Stop locking and use DISTINCT when finding used terms to delete https://gerrit.wikimedia.org/r/547243 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 546925 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@master] Stop locking and use DISTINCT when finding used terms to delete https://gerrit.wikimedia.org/r/546925 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 546925 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [mediawiki/extensions/Wikibase@master] Stop locking and use DISTINCT when finding used terms to delete https://gerrit.wikimedia.org/r/546925 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. Trying to do point number 4, Ran it locally and couldn't find anything we can batch: amsa@amsa-Latitude-7480:/var/log/mediawiki$ tail debug-repo.log --lines 200 | grep DBQuery | grep -v LCStoreDB | grep wbt_ | grep -i insert [DBQuery]

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Jdforrester-WMF added a comment. WikidataCon is happening right now, it's likely that one or more of the cool new tools people have unleashed for Wikidata's birthday are adding a lot of load, or just that overall it's a problem. TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

hashar added a comment. Seems that is occurring again today: F30879681: wikidata_term_deadlock.png https://logstash.wikimedia.org/goto/a6932e800b64c2421f2ba69ed2e39710 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 542915 abandoned by Ladsgroup: mediawiki: Disable query page updates for wikidatawiki Reason: Done in I7d0f973b4416 https://gerrit.wikimedia.org/r/542915 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. At this rate, I think it's fine but if we want to increase the speed or set write_both for all, we should fix the lock time somehow (points 2, 3, and 4) before we move forward TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. In T234948#5579417 , @Ladsgroup wrote: > It's on Q23Mio. Even though the script fails, it logs all of them in mwmaint node, I can get list of all of them and fix them via a script (python to do purge,

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. It's on Q23Mio. Even though the script fails, it logs all of them in mwmaint node, I can get list of all of them and fix them via a script (python to do purge, mwscript, anything) so I don't think that would be a very big deal IMO. These are all secondary data.

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. @Ladsgroup is rebuild script close to Q40m? I'm thinking of: 1. we lowering write both to Q40m, 2. stop rebuild script shortly before it reaches Q40m and 3. investigate what is going on here. But if rebuild script still not that close, we can investigate a bit

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Marostegui added a comment. That's indeed not ideal (the fact that the transaction isn't retried), and probably merits some investigation and some potential mitigation methods as deadlocks can happen anytime (ideally not often, but are a possibility with high concurrency environments).

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. So we checked in new store what terms exist on some of the items that had failing inserts due the dead-locks. None of them, except one, has terms stored on them. Meaning the transactions were not repeated later, and we don't know whether they will fail again.

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Marostegui added a comment. That link doesn't work for me :( TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde, Marostegui Cc: Lydia_Pintscher, alaa_wmde, Jdforrester-WMF, WMDE-leszek,

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. It seems to have improved the situation a lot since yesterday https://logstash.wikimedia.org/app/kibana#/dashboard/87348b60-90dd-11e8-8687-73968bebd217?_g=h@af6022c&_a=h@d6ea75e There are still few spikes, and it seems that they reach a dead-lock only when so

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Marostegui added a comment. I have merged your change and disable `MostLinked` cronjob entry. Thank you! TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde, Marostegui Cc:

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 542956 **merged** by Marostegui: [operations/puppet@production] mediawiki: Split cronjob for updatequerypages to multiple modules https://gerrit.wikimedia.org/r/542956 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 542956 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [operations/puppet@production] mediawiki: Split cronjob for updatequerypages to multiple modules https://gerrit.wikimedia.org/r/542956 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. In T234948#5572680 , @Marostegui wrote: > Another possibility would be to split `modules/mediawiki/manifests/maintenance/updatequerypages/cronjob.pp` into one cron per "feature"? Yeah, on it but it

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Marostegui added a comment. In T234948#5572526 , @Ladsgroup wrote: > In T234948#5569562 , @Lydia_Pintscher wrote: > >> In T234948#5569449

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 542915 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [operations/puppet@production] mediawiki: Disable query page updates for wikidatawiki https://gerrit.wikimedia.org/r/542915 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Lydia_Pintscher added a comment. Ah. Do we have pageview/access stats for these? TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde, Lydia_Pintscher Cc: Lydia_Pintscher, alaa_wmde,

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. In T234948#5569562 , @Lydia_Pintscher wrote: > In T234948#5569449 , @alaa_wmde wrote: > >> It can be that we can actually live, until migration is done,

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Marostegui added a comment. In T234948#5569455 , @alaa_wmde wrote: > Until then, @jcrespo @Marostegui please feel free to kill that script and/or stop running it at all (if possible) when necessary or convenient, and thanks a lot for

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Lydia_Pintscher added a comment. In T234948#5569449 , @alaa_wmde wrote: > It can be that we can actually live, until migration is done, with Most Linked special pages on wikidata not being that up-to-date for about a month? >

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. Until then, @jcrespo @Marostegui please feel free to kill that script and/or stop running it at all (if possible) when necessary or convenient, and thanks a lot for your help figuring this out thus far :) TASK DETAIL https://phabricator.wikimedia.org/T234948

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Marostegui added a comment. I had to finally kill the script again as it was causing issues: https://logstash.wikimedia.org/goto/ecd4d01191b76b075ae00f2477a02d38 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Marostegui added a comment. The script has started again and there's a query that has been running for 4h already, this might end up like yesterday (T234948#5562987 )

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Marostegui added a comment. For the record: https://grafana.wikimedia.org/d/00273/mysql?orgId=1=eqiad%20prometheus%2Fops=db1087=9104=1570660711651=1570716407042=3 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Marostegui added a comment. In T234948#5562953 , @jcrespo wrote: > We found a possible cause even if partial: maintenance queries for Special page update like `SpecialMostLinked::reallyDoQuery` causing lag and general slowdown on other

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

jcrespo added a comment. We found a possible cause even if partial: maintenance queries for Special page update like `SpecialMostLinked::reallyDoQuery` causing lag and general slowdown on other servers, probably you can check what those do and if they break or make the wikidata migration or

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

jcrespo added a comment. Now there seems to be also many duplicate insert errors. TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde, jcrespo Cc: alaa_wmde, Jdforrester-WMF,

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. Nevermind me .. I confused the date in previous comments TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde Cc: alaa_wmde, Jdforrester-WMF, WMDE-leszek,

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. There could be some correlation too with Edit Activity (per single user esp.): https://grafana.wikimedia.org/dashboard/snapshot/uWMkJURalbo79LzeXdDsthh2bZx0Ul2V F30615822: image.png TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. Yeap it is fluctuating .. looking on a very small period of time, consecutive locks happened on consecutive items. https://logstash.wikimedia.org/goto/0f8316d5e336b2bab5a6f88fe0cd2183 There seems to be some correlations with open connections?

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

jcrespo added a comment. It came back https://logstash.wikimedia.org/goto/de0f6611dde37e811edcba4f530131c0 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde, jcrespo Cc: alaa_wmde,

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. This is the last 24 hours : F30614795: image.png It went down to normal but then jumped back again a couple of hours ago. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. I see .. I was fooled by the drop of errors this evening.. must have waited on it a bit esp. knowing that it took longer to publish config changes last time too. Then I'll keep this in Done and open to look at it tomorrow TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. In T234948#5560918 , @alaa_wmde wrote: > Error rate increases again. Problem was not really solved I might be wrong but looking at the logs, it seems it's still writing to Q-id > 70Mio (An example

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-10-09T11:04:44Z] Synchronized wmf-config/InitialiseSettings.php: [[gerrit:541777|Put write both limit down to Q70m for item terms (T234948 )]] (duration: 01m 10s) TASK

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 541777 **merged** by jenkins-bot: [operations/mediawiki-config@master] Put write both limit down to Q70m for item terms. https://gerrit.wikimedia.org/r/541777 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. Deployment of switching the limit for write both is scheduled for today https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20191009T1100 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

gerritbot added a comment. Change 541777 had a related patch set uploaded (by Alaa Sarhan; owner: Alaa Sarhan): [operations/mediawiki-config@master] Put write both limit down to Q70m for item terms. https://gerrit.wikimedia.org/r/541777 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. My hypotheses is that there's a lag for the system to actually pick up the new term store configuration and use it. It happened before that it took five hours until cawiki started to pick up the new config. TASK DETAIL https://phabricator.wikimedia.org/T234948

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. @jcrespo understood thank you Yes we need to understand where it is coming from. @Ladsgroup as it continues to happen till this moment, I think we better either slow down migration or bring down write both to Q40m again. If then the error rate goes down back

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

jcrespo added a comment. > @jcrespo what rate should we be concerned about? (to plan/monitor ahead) As I said above, with the current rate it is not of immediate concern to me, but my suggestion is to understand why it is happening and what are the consequences, to be 100% sure data is

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

alaa_wmde added a comment. Indeed looking at different graphs does not show anything that immediately suggests any reason for this. We can reduce writing to both again to Q40m and see if it is indeed related to that switch (as the actual switch happened ~9hrs earlier than this increase

[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr

Ladsgroup added a comment. There's no change in edit patterns or page creation patterns though: F30592633: image.png It might be some caches start to expire but that's worrying TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL