[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. In T205045#5290536 , @Ladsgroup wrote: > Jobs inserted in the last hour: > F29658135: image.png > Basically it dropped 91% of addEntityUsage jobs ^_^ While

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

gerritbot added a comment. Change 528072 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.16] Add only needed entity usages in AddUsagesForPageJob https://gerrit.wikimedia.org/r/528072 TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

gerritbot added a comment. Change 528072 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.16] Add only needed entity usages in AddUsagesForPageJob https://gerrit.wikimedia.org/r/528072 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

gerritbot added a comment. Change 527131 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@master] Add only needed entity usages in AddUsagesForPageJob https://gerrit.wikimedia.org/r/527131 TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

gerritbot added a comment. Change 527131 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [mediawiki/extensions/Wikibase@master] Add only needed entity usages in AddUsagesForPageJob https://gerrit.wikimedia.org/r/527131 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. In T205045#5382778 , @hoo wrote: > Yeah, this is on purpose… this is rather complex (unfortunately). The thing is that we make sure that usage are "correct" by: Immediately purging no-longer accurate usages

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

hoo added a comment. Yeah, this is on purpose… this is rather complex (unfortunately). The thing is that we make sure that usage are "correct" by: Immediately purging no-longer accurate usages on page touches/ re-parses on edit (it might be that we also add new usages here, but that's not

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. One rather easy way to test my hypotheses is that you can avoid running jobs (by not running runJobs.php) in your localhost and then changing entity usages in pages and seeing `wbc_entity_usage` gets updated accordingly. I did it and it worked confirming my

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

gerritbot added a comment. Change 519518 **merged** by Ppchelko: [mediawiki/services/change-propagation/jobqueue-deploy@master] Lower wikibase-addUsagesForPage concurrency. https://gerrit.wikimedia.org/r/519518 TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Lucas_Werkmeister_WMDE added a comment. In T205045#5288987 , @Ladsgroup wrote: > I will upload the patch soon, just need to write tests For the record, the patch is I45b699a8b8 (already

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

gerritbot added a comment. Change 519518 had a related patch set uploaded (by Ppchelko; owner: Ppchelko): [mediawiki/services/change-propagation/jobqueue-deploy@master] Lower wikibase-addUsagesForPage concurrency. https://gerrit.wikimedia.org/r/519518 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Pchelolo added a comment. In #changeprop for job queue, we have a concept of concurrency - how many jobs of the same type can be running in parallel. `wikibase-addUsagesForPage` had a special rule with concurrency 30, so if this pattern

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. Another graph: F29658175: image.png TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup Cc: Joe, mobrovac,

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Pchelolo added a comment. OMG! ❤️ TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, Pchelolo Cc: Joe, mobrovac, Pchelolo, Michael, Lydia_Pintscher, jcrespo, Ladsgroup, hoo,

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. Jobs inserted in the last hour: F29658135: image.png Basically it dropped 91% of addEntityUsage jobs ^_^ TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. There's even a TODO for it already: https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/client/includes/Hooks/DataUpdateHookHandlers.php#L165 This needs to be cleaned up ASAP. I will take a look later today. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. In T205045#5288620 , @jcrespo wrote: > Yes, we were worried about the frequency of it (it was the most common DB-related error for some time) because of the performance implications, not that it happens

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

jcrespo added a comment. Yes, we were worried about the frequency of it (it was the most common DB-related error for some time) because of the performance implications, not that it happens sometimes. I am guessing that this is due to multiple edits happening at the same time about the same

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. In T205045#5288520 , @Michael wrote: > Could we do something about the error itself? The error message says `Deadlock found when trying to get lock; try restarting transaction` so we could try to catch it,

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Michael added a comment. In T205045#5288488 , @Ladsgroup wrote: > Okay, after decreasing the batch size to 100, the errors have reduced but haven't eliminated, the number is sorta high but I don't know if DBAs think this is bearable or

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. Okay, after decreasing the batch size to 100, the errors have reduced but haven't eliminated, the number is sorta high but I don't know if DBAs think this is bearable or it should be reduced further. TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. In T205045#5268632 , @jcrespo wrote: > Thanks, that looks good. I saw no significant impact impact on db infra, if it does more queries/transactions it had not noticeable issue. Did it impact lag/performance

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

jcrespo added a comment. Thanks, that looks good. I saw no significant impact impact on db infra, if it does more queries/transactions it had not noticeable issue. Did it impact lag/performance on your side? I see a gap here last week-

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. It has improved since the deployment (dropping it to 200): F29604901: image.png TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. In T205045#5225805 , @jcrespo wrote: > > and DBAs to babysit the deployment for a couple of days > > I can do that if you tell me anyting specific -metrics- to check for (other than typical error count and

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

jcrespo added a comment. > and DBAs to babysit the deployment for a couple of days I can do that if you tell me anyting specific -metrics- to check for (other than typical error count and general stability. Specially if initial deployment happens at the beginning of next week for

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. In T205045#5223716 , @Lydia_Pintscher wrote: > @jcrespo thank you! That's useful information. > > @Ladsgroup @Addshore What's needed to get this ready for pickup in camp? It's actually just a config

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Lydia_Pintscher added a comment. @jcrespo thank you! That's useful information. @Ladsgroup @Addshore What's needed to get this ready for pickup in camp? TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. In T205045#4964764, @Addshore wrote: In T205045#4937780, @Ladsgroup wrote: I think so too, in mediawiki, usually it's 100. Is there a legacy reason for us to be using such a high batch size that you are aware of? I'm not aware of any reason whatsoever. It might be

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Addshore added a comment. In T205045#4937780, @Ladsgroup wrote: I think so too, in mediawiki, usually it's 100. Is there a legacy reason for us to be using such a high batch size that you are aware of?TASK DETAILhttps://phabricator.wikimedia.org/T205045EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Ladsgroup added a comment. I think so too, in mediawiki, usually it's 100.TASK DETAILhttps://phabricator.wikimedia.org/T205045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LadsgroupCc: Ladsgroup, hoo, Addshore, Marostegui, Aklapper, Krinkle, Nandana, Lahi,

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Addshore added a comment. In T205045#4878367, @hoo wrote: Maybe EntityUsageTable::INSERT_BATCH_SIZE_FACTOR should be changed from 5 to 3/2.5/2? None of the queries failing with deadlocks seem to be anywhere near the 100*5=500 insert limit, but I guess it's not necessarily the inserts in the

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

hoo added a comment. Maybe EntityUsageTable::INSERT_BATCH_SIZE_FACTOR should be changed from 5 to 3/2.5/2?TASK DETAILhttps://phabricator.wikimedia.org/T205045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: hoo, Addshore, Marostegui, Aklapper, Krinkle,

[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)

Addshore added a comment. Logstash has 835 records matching this error from the past 30 days (at least since 1.32.0-wmf.16). According to my query this was 5,300 for the last 30 days now? If my snapshot & query is correct https://logstash.wikimedia.org/goto/1638470f94494c4efb6557f6e550caae It