jcrespo added a subtask: T178290: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki.
TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, GoranSMilovanovic
jcrespo added a subtask: T12: Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata).
TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To
jcrespo added a parent task: T187521: Optimize recentchanges and wbc_entity_usage table across wikis.
TASK DETAILhttps://phabricator.wikimedia.org/T12EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: gerritbot, Stashbot, Jdforrester-WMF, Bawolff
jcrespo added a comment.
probably commonswiki and ruwiki were done at T12. The list of the ones scheduled to do next was at T178290#3688203TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc
jcrespo added a comment.
Whoever deployed the table, violated the advice when creating new tables: "However, due to labs filtering limitation (hopefully, to be solved soon) you still should block on a DBA to review the table creation to check data leak is not possible.&q
jcrespo added a comment.
Ok, Manuel haven't communicated to me and I wasn't on that ticket. Apparently the tables were created too soon.TASK DETAILhttps://phabricator.wikimedia.org/T190153EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoC
jcrespo added a comment.
+1 in case there are more changesTASK DETAILhttps://phabricator.wikimedia.org/T190153EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, aude, Addshore, Aklapper, Anomie, Tgr, daniel, Lahi, PDrouin-WMF
jcrespo added a comment.
"too soon" in that the structure was not stable, even if it was merged.TASK DETAILhttps://phabricator.wikimedia.org/T190153EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, aude, Addshore
jcrespo removed a project: Data-Services.jcrespo added a comment.
@Yamaha5 I think I know what is the issue- A wikidata edit was done, but the page still have interwiki links, see at the end of: https://en.wikipedia.org/w/index.php?title=Category:Australian_suffragists&oldid=624098262&acti
jcrespo added subscribers: Marostegui, jcrespo.jcrespo added a comment.
@Ladsgroup @Lucas_Werkmeister_WMDE @Marostegui Let's talk here about provinding a temporary wikidata replica so wikidata team can play with changes to it.TASK DETAILhttps://phabricator.wikimedia.org/T188279
jcrespo added a project: DBA.jcrespo added a comment.
Adding #DBA to support the setup of this test db.TASK DETAILhttps://phabricator.wikimedia.org/T188279EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, Nikerabbit
jcrespo added a comment.
We can either change the password, or the user, or both.
+1TASK DETAILhttps://phabricator.wikimedia.org/T188279EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Stashbot, gerritbot, jcrespo, Marostegui, Nikerabbit
jcrespo removed a project: DBA.jcrespo added a comment.
Wikidata team will test it on the test host and create a task for production deployment with all requested changes/full strategy. Until then, there is nothing for us to do here. If you need help, re-add us with a specific request.TASK
jcrespo added a comment.
I found T86530, which may be outdated, but may help with giving more options.
As a note to @Ladsgroup, and the other people working on this- after dropping a column, replication stopped working on the test host (as expected). This is not a problem, just a remark so you
jcrespo added a comment.
commonswiki errors due to deadlocks on INSERT IGNORE INTO wbc_entity_usage seem to be common (not too worrying, but on of the most comon database errors), could the code be optimized to avoid those? I am guessing that the same row is written many times (once per change on
jcrespo created this task.jcrespo added projects: Wikidata, Wikimedia-log-errors.Herald added a subscriber: Aklapper.
TASK DESCRIPTIONSeparating from T187521#4133553
commonswiki errors due to deadlocks on INSERT IGNORE INTO wbc_entity_usage seem to be common (not too worrying, but on of the most
jcrespo added a comment.
Done as T192349TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, GoranSMilovanovic, Marostegui, Aklapper, revi, Lydia_Pintscher, Ladsgroup, Lahi, Gq86, QZanden
jcrespo added a comment.
Depooled db1109 2 days ago, I still cannot put it under maintenance.
I still see it on other sections other than s8, see T143870.
If the code doesn't work, special mediawiki configuration should be setup so that dump hosts only know about dump dbs, I can see that wo
jcrespo added a comment.
@Ladsgroup after the table rebuilding on dbstore2001, purging is going at a good pace:
https://grafana.wikimedia.org/dashboard/db/mysql?panelId=11&fullscreen&orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=dbstore2001&var-port=13318&from=152468
jcrespo added a project: Wikidata.jcrespo added a comment.
Most of the warnings (I suppose the logging was changed to understand which component relates to) seem to be related to wikibase client jobs right now:
https://logstash.wikimedia.org/app/kibana#/dashboard/DBQueryTASK DETAILhttps
jcrespo added a comment.
Full table scans are indeed expected and preferred when tables have 0 to a only a few rows. However, I think Bstorm is right to be concerned, we have seen lately an increase in load on labsdbs, but I have yet to check if it is related to the usage of views or just
jcrespo added a comment.
The indexes seem disproportionally large compared to the data. Could the table be split somehow in a 1:1 relationship that could make sense (e.g. index_table with just columns to search and data_table with almost just the primary index).TASK DETAILhttps
jcrespo added a comment.
Yes, that is much better.TASK DETAILhttps://phabricator.wikimedia.org/T191391EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Lahi, Gq86
jcrespo added a comment.
This would have been very useful in this case https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.htmlTASK DETAILhttps://phabricator.wikimedia.org/T194270EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Marostegui
jcrespo added a comment.
Most if not all of these seem to have gone since yesterday's train (need more time to check if absolutly all are gone or just the bulk of it).TASK DETAILhttps://phabricator.wikimedia.org/T191282EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/
jcrespo lowered the priority of this task from "High" to "Normal".jcrespo added a comment.
@mmodell Most of the errors are gone, but some are still happening. I think this is no longer high, but if it has an easy fix, those should be looked at by the code owners:
https://lo
jcrespo added a comment.
I am not too worried about exceptions/error messages, I only pointed those in case it helped debug the real issues, the ones I mentioned at T198049#4310282 (in first appearance caused by a single production host having issues).TASK DETAILhttps://phabricator.wikimedia.org
jcrespo added a comment.
Let's keep an eye on performance metrics, specially https://grafana.wikimedia.org/dashboard/db/save-timing Although with the estimations given T183488#4480579, it may be desirable to make things faster rather and short than extended for a long period of time.
Meanwhi
jcrespo reopened this task as "Open".jcrespo added a comment.
dumpJson.php is failing many times per minute, while I only rebooted for maintenance an rc replica (which is fully depooled):
https://logstash.wikimedia.org/goto/e2dee89b02bc72774c4320671d879b05TASK D
jcrespo reopened subtask T147169: Make sure Wikibase dump maintenance scripts solely use the "dump" db group as "Open".
TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Lydia_Pi
jcrespo added a comment.
does not try to refresh it periodically
But db1101 never was a dump host, but rc, and it used to have load 1.TASK DETAILhttps://phabricator.wikimedia.org/T147169EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc
jcrespo added a comment.
No need to be sorry, could this be a load balancer bug?TASK DETAILhttps://phabricator.wikimedia.org/T147169EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Lucas_Werkmeister_WMDE, ArielGlenn, thiemowmde
jcrespo added a comment.
@tstarling Please stop writes going to *s2* unless they have already finished, there are plans to put s2 in read only without much pre-warning (unscheduled read only) T201694- I don't know exactly what is going to be the plan (a logical failover or a network mainte
jcrespo added a comment.
I still have no idea how this could happen (also these errors don't seem to be on mwlog1001?!).
Does dumpJson.php call any function that could force any check on all replicas to fail (and because it has old configuration, it fails on a depooled non-slow host
jcrespo added a comment.
Related: T135851 T164488#3530600 T162593#3175313 T162807#3882309 T163190#3246365 T160509#3110389TASK DETAILhttps://phabricator.wikimedia.org/T202032EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: Aklapper, daniel
jcrespo added a comment.
I mentioned this to Tim, and he said to better mention it to Brad (and Daniel?), that it would be nice to run a read-only process to check the consistency of the migration- it should take much less time than the writes and would avoid headaches given the issues with the
jcrespo added a comment.
In T183488#4506824, @daniel wrote:
We did pretty extensive testing of this on the test copies (db and db1112). We could have a script that does basic sanity checks, but I'm not sure what to test, beyond the trivial. All I can think of is:
all revision rows ha
jcrespo added a comment.
So the script will make sure to not miss any content, but doesn't assure it introduces extra one, or duplicates some, that is why I mention the usage of extra checks. Takes very little time (1 long running query per check) and validates the whole process. The same
jcrespo updated the task description. (Show Details)jcrespo added a project: Wikidata.
CHANGES TO TASK DESCRIPTIONWe are seeing lots of errors related with the following output (https://logstash.wikimedia.org/goto/f8ec4b3e22a8b3d5a8ad0143a4f8982f):
```
server:www.mediawiki.org
jcrespo added a comment.
beta has a single shard
s/shard/section/TASK DETAILhttps://phabricator.wikimedia.org/T202483EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Stashbot, gerritbot, Addshore, Anomie, daniel, tstarling, Paladox
jcrespo added a comment.
@Lucas_Werkmeister_WMDE What is the plan with constraint_guid? Will you drop it? Shouldn't it keep the UNIQUE constraint while it is in use (and by extension, the index)? We need a quick answer if this is to be done in September.TASK DETAILhttps://phabricator.wikimedi
jcrespo added a comment.
Please if you could give this a lot of priority, because if we miss this "train" (switchover) we could be waiting for a full extra year. Thanks. PK changes are not easy to do normally.TASK DETAILhttps://phabricator.wikimedia.org/T189101EMAIL PREFER
jcrespo added a comment.
I got this on mediawiki logs at the same time than one of the retries. Are you sure you are not using a deprecated query? I don't know if they are correlated, but it happened in the same time frame and it is json related.
Warning: API call had warnings trying t
jcrespo added a comment.
@Ladsgroup That could be a separate issue and could be handled on a separate task (I don't know). The on topic here is UsageAspectTransformer.php (probably you saw that, just in case my comment was misleading about what was the main actionable here).TASK DETAIL
jcrespo added a comment.
Thank you anomie for running those queries, which I had suggested or even offered to do to check things look consistent.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling
jcrespo added a comment.
Yeah, let's announce it first as this time is a "breaking change" and we can wait a bit to deploy. I have also pending to to a rolling restart for mariadb upgrade and we can do it at the same time as otherwise for large tables all changes may fail beca
jcrespo added a comment.
will anything break
I don't think because fields have been added but not yet removed, but I cannot promise that will happen soon or it is happening now (we should test on deploy on a smaller db first).TASK DETAILhttps://phabricator.wikimedia.org/T174047
jcrespo added a comment.
I've written this: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Stability_of_the_mediawiki_database_schema please review if it makes sense and ping if you see issues or edit it yourself and it would be great if you could send an email referencing th
jcrespo added a comment.
@daniel look at the writeup I mention above, and see if it seems reasonable- note that tools may rely on old structure, in terms of fields may not be dropped yet, but I guess they will eventually be emptied. Note that the main issues we learned is that compatibility views
jcrespo added a comment.
no issue with scwiki or any other small one- you may still need he depooling to avoid headaches on the larger ones :-)TASK DETAILhttps://phabricator.wikimedia.org/T174047EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc
jcrespo created this task.jcrespo added projects: Wikimedia-production-error, Dumps-Generation, Wikidata, MediaWiki-Logging.
TASK DESCRIPTION(Not sure yet if these are the regular dumps or the wikidata-specific ones)
Lots of errors coming from snapshot1008 related to replication lag and read only
jcrespo updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTION...While this didn't cause issues to final issues, this cause some issues on production due to impact to the logging infrastructure: https://grafana.wikimedia.org/dashboard/db/production-logging?orgId=1
jcrespo added a project: Datacenter-Switchover-2018.
TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Marostegui, Banyek, Reedy, gerritbot, Krinkle, Addshore, Yurik, jcrespo, Imarlier, Ladsgroup
jcrespo added a comment.
The plan is for us dbas to test setting up a single API with the same structure than eqiad and do all assuming that fixies it, and later we will have to evaluate what is the right long-term status, given some unknowns and related tasks such as T202167:
One note: This
jcrespo added a project: Datacenter-Switchover-2018.jcrespo added subscribers: akosiaris, Joe.jcrespo added a comment.
CC @Joe @akosiaris not to solve the specific issue, but to note it as a potential missed step/puppet code change related to switch datacenter.
Having said that, if dumps are run
jcrespo changed the status of subtask T172489: Monitor read_only on all databases, make it page on masters from "Open" to "Stalled".
TASK DETAILhttps://phabricator.wikimedia.org/T171928EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcresp
jcrespo moved this task from Triage to Blocked external/Not db team on the DBA board.jcrespo added a comment.
^I am assuming this doesn't have yet actionables for us (yet) just on the radar.TASK DETAILhttps://phabricator.wikimedia.org/T204838WORKBOARDhttps://phabricator.wikimedia.org/project/
jcrespo added a comment.
Maybe related or blocker: T106363
I am interested on this happening at some point, I just saw lots of revision data (not metadata) stored on the text table and its impact on a quick database recovery.TASK DETAILhttps://phabricator.wikimedia.org/T183490EMAIL
jcrespo added a comment.
We know the exact timestamps of missing rows,
from
db1071-bin.007238:795791989 2018-09-13 09:08:17 on the active (codfw) master: db2045-bin.005879:1036765620
to
db1071-bin.007238:796727644 2018-09-13 09:58:26 on the active (codfw) master: db2045-bin.005880:132937482
We
jcrespo added a comment.
applied a fix to:
db1109
db1071
db1104
db1101:3318
db1092
dbstore1002TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark
jcrespo added a comment.
We believe the most important issues (missing revisions, pages and users, and making accessible the content) has been fixed. We will lower the UBN after doing some additional sanity checks. Having said that, for the following days, some missing things like user preferences
jcrespo added a project: User-notice.jcrespo added a comment.
For #user-notice, if it seems reasonable, see T206743#4658525 for the state of things- data was temporarily lost (for around 1 day) but then it was recovered once detected. There may be smaller issues (preferences lost, what links here
jcrespo lowered the priority of this task from "Unbreak Now!" to "High".jcrespo added a comment.
High because data came back.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui,
jcrespo added a comment.
@Pigsonthewing Please don't get worry- as I said at T206743#4658620 only primary data was recovered, cache tables may show still wrong data until we fully recover everything this week, showing some weird data. Please be patient, but don't touch anything that s
jcrespo added a comment.
@Addshore The main task now is to check pages and pages from revisions that were lost and then restored, and check edits done since last wednesday on those pages, as, while the data has not been lost, those may have put an edit over an older version as the latest, hiding
jcrespo added a comment.
I'll generate a list once all slaves that serve mw traffic are re imaged :)
This is true right now for all pooled read only replicas, but I need to fix the master.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedi
jcrespo added a comment.
@jcrespo Do you have any more information for what to look out here?
The problem is most likely the loadbalancer- working with old data, even if the server is not needed, generates lag warnings for some complex dependency of how lag check works (generating 40K logs per
jcrespo added a comment.
From another thread, from TimS:
It sounds like the snapshot hosts were in fact needing and using the host. It's not LoadBalancer's fault if some maintenance script calls wfGetDB() without specifying a query group. Throwing an exception is the correct thing to do
jcrespo added a comment.
After many fixes during the weekend, wb_terms also fixed on labs, all hosts should be consistent now, doing some extra checks to verify everything is good.
@Addshore did you sent to wikidata users the list you compiled to check bot activity?TASK DETAILhttps
jcrespo lowered the priority of this task from "High" to "Normal".jcrespo added a comment.
We believe this to be fixed fully both wikireplicas and on production, but will not close until extra checks confirm so.TASK DETAILhttps://phabricator.wikimedia.org/T206743EM
jcrespo added a comment.
@Addshore I thought you had communicated to wikidata users about that? Apparently not, or @Pigsonthewing didn't see it, could you link your messages to him?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/
jcrespo added a comment.
@Pigsonthewing I hope my comment at Wikidata Village Pump was helpful- if you think that is ok, I would suggest closing this task, and open a different one to track the merges of old history (this was to track the recovery from backups)?TASK DETAILhttps
jcrespo moved this task from In progress to Done on the DBA board.jcrespo added a comment.
In particular, I see https://www.wikidata.org/wiki/Q2058295 properly merged.TASK DETAILhttps://phabricator.wikimedia.org/T206743WORKBOARDhttps://phabricator.wikimedia.org/project/board/1060/EMAIL
jcrespo added a comment.
loading a table with ROW_FORMAT=COMPRESSED but I am not sure compression is actually the cause, but maybe (I would actually be more sure about that) just a simple rebuild from scratch after a many edits are suffered (something common after >1000M edits).
Compression
jcrespo added a comment.
it got missed somehow
How can that be?, the table was empty or clean at that time according to my notes-and this was verified by 2 people independently (Manuel and me) on 2 different servers as I have on my notes on the 16 oct, and later at T206743#4691048 , unless I am
jcrespo added a comment.
Could you check the list of schema changes and maintenance to be ran during switchover to test if they where also undone?TASK DETAILhttps://phabricator.wikimedia.org/T203709EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui
jcrespo added a comment.
To be fair, technically, this is resolved because wb_terms has been replaced,
AFAIK, with a more specialized mechanism (several smaller and normalised
tables) :-)
TASK DETAIL
https://phabricator.wikimedia.org/T86530
EMAIL PREFERENCES
https
jcrespo closed subtask T165756: Create summary templates on Wikitech wiki to
stop writing the same things everywhere, everytime as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T165726
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailprefer
jcrespo added a comment.
@hoo you are right, but check my comment
<https://phabricator.wikimedia.org/T101502#1341481> on a very similar issue on
https://phabricator.wikimedia.org/T101502.
We need to validate that change on all queries from the same piece of code (not
only for this part
jcrespo added a subscriber: jcrespo.
jcrespo added a comment.
First comments from a MySQL DBA, assuming that goes to production:
- Some tables lack of a `PRIMARY KEY`.** This should be a requirement for new
tables**.
- character strings do not have a specified `CHARACTER SET`. Check that those
jcrespo added a blocked task: T17441: Some tables lack unique or primary keys,
may allow confusing duplicate data.
TASK DETAIL
https://phabricator.wikimedia.org/T102992
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, Tamslo
jcrespo added a subscriber: jcrespo.
TASK DETAIL
https://phabricator.wikimedia.org/T107602
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Joe, jcrespo
Cc: jcrespo, Legoktm, gerritbot, Smalyshev, BBlack, Joe, daniel, RobLa-WMF,
Aklapper, aude
jcrespo added a comment.
By @hoo request, I have applied the schema change also to testwikidatawiki.
Please, in the future, for us silly DBAs :-), we need affected wikis very
explicitly.
TASK DETAIL
https://phabricator.wikimedia.org/T99459
EMAIL PREFERENCES
https
jcrespo added a subscriber: jcrespo.
jcrespo added a comment.
This query, which has to be executed in a serialized way due to replication, is
causing a lag problem on some of the less powerful database slaves (db1047),
and extra load on the rest of the servers. This is specially true for s3
jcrespo added a comment.
@aude If you are increasing the priority based on my comment, don't. Regular
databases can handle the extra load (for now), the main issue is in research
databases or labs, where they contain all dbs, or when there is maintenance and
we are in a degraded state.
jcrespo raised the priority of this task from "High" to "Unbreak Now!".
jcrespo added a subscriber: ori.
jcrespo added a comment.
Sorry, reverting back to Unbreak now, because this is breaking almost all wikis
and the problems have been spoted now on production machines,
jcrespo lowered the priority of this task from "Unbreak Now!" to "High".
jcrespo added a comment.
This is the lag report of db1028.
F1511025: Screenshot from 2015-08-15 11:30:15.png
<https://phabricator.wikimedia.org/F1511025>
Seems to have calmed for
jcrespo added a comment.
And here it is actual data of an example spike:
Host UserSchema Client Source Thread Transaction Runtime
Stamp
db1038wikiusersimplewiki mw1001 - 5096510924
30A18B0D6 15s 2015-08-15 10:04:05
UPDATE
jcrespo added a comment.
Could this job have caused:
/* JobRunner::commitMasterChanges 127.0.0.1 */
GET_LOCK('jobrunner-serial-commit', 30) AS lockstatus /*
31bdf25cc42f95f4f4a9b493cafe0299 db1024 plwiki 11s */ /* localhost */
we had another large lag spike around the same time
jcrespo added a comment.
I am almost sure that the first part of my previous comment was not caused by
this job, but a very large LinksUpdate::incrTableUpdate, sorry about that.
TASK DETAIL
https://phabricator.wikimedia.org/T107319
EMAIL PREFERENCES
https://phabricator.wikimedia.org
jcrespo added a comment.
@JanZerebecki not sure if you are asking me: Yes, IF that is the case (I cannot
say)- batching will actually be worse than no batching at all- as it will be
slower than a single query and more open to clashing with other queries without
any of the advantages (reduced
jcrespo created this task.
jcrespo added a subscriber: jcrespo.
jcrespo added projects: Wikidata, MediaWiki-Database.
Herald added a subscriber: Aklapper.
TASK DESCRIPTION
There seems to be a large number of API calls to wikidata creating a high
number of deadlocks like this (450/hour
jcrespo closed this task as "Declined".
jcrespo claimed this task.
jcrespo added a comment.
I'm more than ok with the work you are doing, I was just suggesting making the
code faster :-)
TASK DETAIL
https://phabricator.wikimedia.org/T111535
EMAIL PREF
jcrespo added a subscriber: jcrespo.
TASK DETAIL
https://phabricator.wikimedia.org/T111769
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, aude, Aklapper, Wikidata-bugs
___
Wikidata
jcrespo added a blocked task: T30599: Deadlock tracking bug (tracking).
TASK DETAIL
https://phabricator.wikimedia.org/T111535
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: aude, JanZerebecki, Addshore, Aklapper, jcrespo, Wikidata-bugs
jcrespo triaged this task as "Low" priority.
TASK DETAIL
https://phabricator.wikimedia.org/T111535
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: aude, JanZerebecki, Addshore, Aklapper, jcrespo, Wik
jcrespo added a comment.
I set the priority to low "nice to fix it, but it is not breaking anything
now". Feel free to disagree with me.
TASK DETAIL
https://phabricator.wikimedia.org/T111535
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
T
jcrespo closed blocking task Restricted Task as "Invalid".
TASK DETAIL
https://phabricator.wikimedia.org/T111954
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Smalyshev, Karima, jkroll, Wikidata-bugs, Jdouglas, aud
jcrespo removed a blocking task: Restricted Task.
TASK DETAIL
https://phabricator.wikimedia.org/T111954
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Smalyshev, Karima, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana,
Manybubbles
jcrespo added a comment.
It could be an enqueue issue, if thousands of those queries are sent at the
same time (due to bot traffic, for example), they would be enqueued on server
side and executed with controlled concurrency. I could check the binary logs to
see if that is the case, and
201 - 300 of 588 matches
Mail list logo
