jcrespo moved this task to Backlog on the Database workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T111769
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: daniel
jcrespo moved this task to Done on the Blocked-on-schema-change workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T62539
WORKBOARD
https://phabricator.wikimedia.org/project/board/1494/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
jcrespo moved this task to Done on the DBA workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T62539
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Stashbot, hoo
jcrespo added a comment.
Please confirm the change is correct, then resolve:
MariaDB [wikidatawiki]> SHOW CREATE TABLE wb_terms\G
*** 1. row ***
Table: wb_terms
Create Table: CREATE TABLE `wb_terms` (
`term_row_id` bigint
jcrespo added a comment.
Yesterday at ~19:00UTC there was an increase 2x in the number of UPDATES on
almost all wikis. I tracked this down to large amounts of `UPDATE /*
Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsageBatch` queries. Is there
anything that could have caused that? What
jcrespo closed this task as "Resolved".
jcrespo added a comment.
One week without feedback, closing as resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T62539
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Sta
jcrespo added a comment.
Marius, or someone else, do you know if this is still ongoing after latest
deployments + hardware upgrades?
TASK DETAIL
https://phabricator.wikimedia.org/T116404
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc
jcrespo lowered the priority of this task from "High" to "Low".
jcrespo placed this task up for grabs.
jcrespo added a comment.
I promise I will give it a thorough check before trying to resolve it, but
let me lower the priority if it is not creating a noticeable impact.
jcrespo added a project: Wikidata.
jcrespo added a comment.
I think the maxlag parameter for bots made it worse, not better.
TASK DETAIL
https://phabricator.wikimedia.org/T135471
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc
jcrespo edited projects, added MediaWiki-General-or-Unknown; removed
MediaWiki-extensions-General-or-Unknown.
jcrespo added a comment.
It could be related to https://phabricator.wikimedia.org/T135470, but the
queries were being locked on a non-API node, so I assume they are only a result
of
jcrespo added a comment.
I think there are several issues with more than one bot, mostly coming from
the loss of database capacity mentioned in
https://phabricator.wikimedia.org/T135100#2289175. Whatever were "bad bot
behaviours" before were masked by the overcapacity. Now t
jcrespo added a comment.
Based on other errors, it could be batches of CategoryMembershipUpdates but
just based on time frame (no conclusive evidence). It could be they just are
affected at the same time.
TASK DETAIL
https://phabricator.wikimedia.org/T135471
EMAIL PREFERENCES
https
jcrespo added a comment.
I think I found it:
| COMMIT /* DatabaseBase::deadlockLoop BotNinja */
|0.000 |
| NULL
|0.000
jcrespo added a comment.
This bot was doing 10 edits per second, causing lag, specially on
recentchanges on both wikidata and dewiki, and probably 10 in parallel. API
policy <https://www.mediawiki.org/wiki/API:Etiquette> does not set a concrete
limit, but it is very clear about not be
jcrespo added a comment.
A visual comparison:
F4022609: ninja.png <https://phabricator.wikimedia.org/F4022609>
TASK DETAIL
https://phabricator.wikimedia.org/T135471
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Lydia_Pin
jcrespo added a comment.
Independently of the policy, which says:
> "There is no hard and fast limit on read requests, but we ask that you be
considerate and try not to take a site down. Most sysadmins reserve the right
to unceremoniously block you if you do endanger the stab
jcrespo added a comment.
Termininja, let's try to fix this. Can you run edits exclusively in a serial
way, as suggested, even if that means they will be slower?
TASK DETAIL
https://phabricator.wikimedia.org/T135471
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/
jcrespo added a comment.
Termininja, you do not need to write only once per minute, something like 1
POST/write per second is unlikely to cause issues, just wait for one edit to
finish before starting the next one (DO NOT SENT MULTIPLE REQUEST IN PARALLEL).
I will tell the admins on
jcrespo closed this task as "Resolved".
jcrespo claimed this task.
jcrespo added a comment.
Your account has been already unblocked.
TASK DETAIL
https://phabricator.wikimedia.org/T135471
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To:
jcrespo created this task.
Herald added subscribers: Zppix, Aklapper.
TASK DESCRIPTION
There is a huge amount of connections to the master, as many that there are
5000 connection errors per minute.
While I cannot yet put a reason to it, most of the connections that fail seem
to be
jcrespo edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Aklapper, jcrespo, Zppix, Minhnv-2809, Volans, D3r1ck01, Izno, Luke081515,
Wikidata-bugs, aude
jcrespo edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: JanZerebecki, hoo, aude, Aklapper, jcrespo, Zppix, Minhnv-2809, Volans,
D3r1ck01, Izno, Luke081515
jcrespo added a comment.
There are a couple of long-running tasks on the master, from snapshot1003-
that is strange, but not sure it is an issue.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo added a comment.
Sorry I wasn't specific enough. This issue is only happening on db1049:
s5-master (Wikidata master).
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcres
jcrespo added a comment.
If you ask for wikis, this is the distribution:
filter: `wikibase-addUsagesForPage and 10.64.16.144`
(10.64.16.144 is s5-master)
dewiki (777) enwiki (691) wikidatawiki (487) frwiki (423) zhwiki (415)
commonswiki (384) svwiki (375) itwiki (287) ptwiki
jcrespo added a comment.
A high sustained level of errors seem to have gone at 21:40-21:45 yesterday.
The errors continue, but only in smaller spikes. We'll see if it returns when
user activity grows more.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
jcrespo added a comment.
I do not know what happened there:
F4098270: Screenshot from 2016-06-01 11:36:39.png
<https://phabricator.wikimedia.org/F4098270>
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings
jcrespo added a comment.
@JanZerebecki - our max_connections is very high: 1 simultaneous
connections- because we use connection pooling at server side. We almost never
reach that- mediawiki servers get saturated first; what fails is the connection
timeout, which is set to 3 seconds
jcrespo reopened this task as "Open".
jcrespo added a subscriber: ArielGlenn.
jcrespo added a comment.
This issue happened again today between 4:26 and 7:20 (I may have done
something to end it because I had to do some unrelated server changes):
https://logstash.wikimedia.org/
jcrespo moved this task from Triage to In progress on the DBA board.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hoo, jcrespo
jcrespo added a comment.
@hoo, my apologies- I thought it had been already deployed. Let me keep it
open until it deploys so that 3rd parties can see it can happen for now- you
can move it if you need it on the
https://phabricator.wikimedia.org/tag/wikidata/ dashboard.
TASK DETAIL
https
jcrespo added a comment.This is not a problem with the servers, the query planner, or the indexing:
MariaDB db1068 commonswiki > EXPLAIN SELECT DISTINCT eu_entity_id FROM `wbc_entity_usage` WHERE eu_entity_id IN ('Q148475','Q54919','Q423048','Q
jcrespo added a comment.@daniel I think this is a case of prematurely optimizing. It is true that things like:
foreach ... { 'SELECT' }
Are usually considered bad practices, but for trying to minimize "round-trip time", we are actually doing a way worse query. Your query c
jcrespo added a comment.A slight comment here: I know the second query is way better than the the first one- there could be an even better solution; but just doing this change would be a huge improvement (at least 100x faster).TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL
jcrespo added a comment.I like your approach @daniel, starting simple, then optimize with a more complex option. Thank you!TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Zppix
jcrespo closed this task as "Resolved".jcrespo added a comment.
Master connection issues not seen for a week.TASK DETAILhttps://phabricator.wikimedia.org/T136598EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: Stashbot, ArielGle
jcrespo created this task.jcrespo added projects: Performance, Wikidata.
TASK DESCRIPTIONHi, with the resolution on the previous issue (T137539), performance has increased, however, I have detected an increase on the number of temporary tables:
F4172090: Screenshot from 2016-06-16 17:49:41.png
jcrespo edited the task description. (Show Details)
EDIT DETAILS...This maybe circumstantial, and not even related, but there is in some cases a slight performance improvement while using `UNION ALL` instead of `UNION`, the first not neededing a temporary table and being able to return rows
jcrespo created blocking task T137972: Use UNION ALL instead of UNION for bulk wbc_entity_usage queries.
TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Zppix, Aklapper, hoo, aude
jcrespo added a project: Wikidata.jcrespo added a comment.
This is not only happening for dumps, terbium is also wrongly using main-dbs (which are still on testing) for long-running queries, which cause long periods of connection issues: https://logstash.wikimedia.org/#dashboard/temp
jcrespo added a comment.
@ArielGlenn : let's identify the reason why this is happening before changing things- we may implement proxying/etcd config before changing any logic. The important part here and now is that probably some mediawiki class is not using the dump role, maybe relat
jcrespo added a subscriber: hoo.jcrespo added a comment.
With 'dump', I sometimes mean 'vslow', too (e.g. for terbium).TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: hoo,
jcrespo added a comment.
Probably related to T138208, I would set that as unbreak now.TASK DETAILhttps://phabricator.wikimedia.org/T138291EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Stashbot, hoo, Melderick, Zppix, D3r1ck01, Izno
jcrespo added a comment.
Right now my only worries are for wikidata, because they create large amount of connection errors and are very visible- I can check on other shards, but even if they do, it would be very low priority (no infrastructure issues there).TASK DETAILhttps
jcrespo added a comment.
10.64.48.26 is db1071, a regular-traffic db.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude
jcrespo added a comment.
These jobs need access to the main DBs
What is a "main db" and what is the difference with a 'vslow' slave? You are accessing a testing slave, that will be put down at any moment (or will block & kill terbium traffic).TASK DETAILhttps://p
jcrespo added a comment.
As I said:
If these create light-weight queries only, lets disconnect and connect after some amount of seconds. If dump hosts are too slow, let's give them better resources. If there is a problem with the connection framework, let'x fix it with the addition
jcrespo added a comment.
Let me show you the weight of these servers:
's5' => array(
'vslow' => array(
'db1045' => 1,
),
'dump' => array(
'db1045' => 1,
),
'api' => array(
'db1070' =>
jcrespo added a comment.
However, there are potentially several dozen places where we call LoadBalancer::getConnection (we try not to hog the connection, but only get it from the LB when we need it - so we do that often). We'd have to somehow loop this parameter through to all the places whe
jcrespo added a comment.
This is not an emergency, I handled that, but it should be definitely 'high'- I suspect it is what it is causing queries such as dumps fail/go slow. T138291 has a different root cause, but I assume it is related to this.
There has been dewiki bot (api) users c
jcrespo added a comment.
@Dzahn check your mail.TASK DETAILhttps://phabricator.wikimedia.org/T134017EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Jdforrester-WMF, RobH, Lydia_Pintscher, StevenJ81, hoo, aude, Multichill, VIGNERON, Nikki, chasemp
jcrespo added projects: DBA, Wikidata.
TASK DETAILhttps://phabricator.wikimedia.org/T139618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Magnus, Aklapper, jcrespo, Zppix, Minhnv-2809, Volans, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude
jcrespo moved this task from incoming to monitoring on the Wikidata board.
TASK DETAILhttps://phabricator.wikimedia.org/T139618WORKBOARDhttps://phabricator.wikimedia.org/project/board/71/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: yuvipanda
jcrespo added a comment.
@Magnus, as you can see on the discussion I agreed with you initially, and in no way I am giving you any responsibility for this particular incident. However, the user says he doesn't have the *option* of running it slower, in a single thread, if I understood corr
jcrespo moved this task from Triage to Blocked external/Not db team on the DBA board.jcrespo added a comment.
I will see what the user responds, and act depending on it.TASK DETAILhttps://phabricator.wikimedia.org/T139618WORKBOARDhttps://phabricator.wikimedia.org/project/board/1060/EMAIL
jcrespo added a comment.
I think we agreed to use only one "tab" at a time to follow API:Etiquette. I will block all your queries if they continue producing errors in the next 10 minutes, as I have warned you 3 or 4 times.TASK DETAILhttps://phabricator.wikimedia.org/T139618EMAIL PREFER
jcrespo created this task.jcrespo added projects: Wikidata, MediaWiki-Database.Herald added subscribers: Zppix, Aklapper.
TASK DESCRIPTIONLots of errors while executing:
INSERT INTO `wb_entity_per_page` (epp_entity_id,epp_entity_type,epp_page_id,epp_redirect_target) VALUES ('25615547&#
jcrespo closed this task as "Resolved".jcrespo claimed this task.jcrespo added a comment.
Thank you. I see lower amount of errors in the last 20 minutes. I will be monitoring the logs in case the errors return.TASK DETAILhttps://phabricator.wikimedia.org/T139618EMAIL PREFER
jcrespo changed the title from "Petscan is running too fast for Wikidata" to "Petscan is being used with excesive parallelism by a user on Wikidata".
TASK DETAILhttps://phabricator.wikimedia.org/T139618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailprefe
jcrespo added a comment.
Low priority for this right now (the user war editing with a parallelism of 35), but maybe a problem in the future if the edition growth keeps going up.TASK DETAILhttps://phabricator.wikimedia.org/T139636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
jcrespo reopened this task as "Open".
TASK DETAILhttps://phabricator.wikimedia.org/T139618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: zhuyifei1999, Jura1, Bugreporter, Sjoerddebruin, yuvipanda, Magnus, Aklapper, jcrespo, Zppix, M
jcrespo added a comment.
This is still ongoing.TASK DETAILhttps://phabricator.wikimedia.org/T139618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: zhuyifei1999, Jura1, Bugreporter, Sjoerddebruin, yuvipanda, Magnus, Aklapper, jcrespo, Zppix, Minhnv
jcrespo changed the title from "Database contention under high edit rate" to "Wikidata Database contention under high edit rate".
TASK DETAILhttps://phabricator.wikimedia.org/T139636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcresp
jcrespo added a comment.
Since 10:30 the number of log errors has increased to 2000/hour, probably related to Wikidata API requests by bots.TASK DETAILhttps://phabricator.wikimedia.org/T140955EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo
jcrespo added a comment.
This (although it is not the only one) is causing problems with huge sizes on logs. I would suggest to either disable the logging or sample them or all logs will be broken!
Then you can continue fixing it in the same way before put it back to error level.TASK DETAILhttps
jcrespo added a comment.
I will check the logs.TASK DETAILhttps://phabricator.wikimedia.org/T140955EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: gerritbot, hashar, aaron, hoo, Lydia_Pintscher, daniel, aude, jcrespo, Aklapper, TerraCodes, greg
jcrespo added a comment.
Indeed, it continues happening on WMF-servers. As we have the tag #wikimedia-log-errors (a WMF issue rather than a mediawiki or wikidata-only), I would like to keep this open if you are ok with that to check, for example, it does not happen by other unrelated reasons (both
jcrespo added a comment.
I cannot see this anymore since 23 UTC yesterday. If you can confirm its successful deployment, I will mark this as resolved.TASK DETAILhttps://phabricator.wikimedia.org/T140955EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc
jcrespo added subscribers: Manuel, jcrespo.jcrespo added a comment.
Yes, a dedicated database service for this could make sense to separate writes if tracking changes starts taking most of the database writes. I suggested this as a means to offload some of the load if it starts to become a problem
jcrespo raised the priority of this task from "Normal" to "Unbreak Now!".jcrespo added a comment.
The execution of:
www-data 20284 0.0 0.1 328216 60744 ?S15:30 0:00 php5 /srv/mediawiki-staging/multiversion/MWScript.php extensions/Wikidata/extensions/Wikib
jcrespo raised the priority of this task from "High" to "Unbreak Now!".Herald added subscribers: Liuxinyu970226, Jay8g, TerraCodes.
TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/T
jcrespo added a comment.
I think this is just UPDATE /* Title::invalidateCache */ based on the binlogs, not the above script.TASK DETAILhttps://phabricator.wikimedia.org/T173269EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: WMDE-leszek, jcrespo
jcrespo added a comment.
Setting as unbreak now because this is preventing collaborators from editing articles, which causes not only to slowdown volunteers, also causing large frustration among them: https://grafana.wikimedia.org/dashboard/db/wikidata-edits?refresh=1m&panelId=1&fullscreen
jcrespo updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTION**Edit**: This used to be punctual at some point, now it is causing generalized edit slowdowns and inability to read recentchanges/watchlists on several wikis.
The most impacted ones are s2 and s7, but looks
jcrespo added a comment.
which means most probably a direct cause of T164173, getting worse?TASK DETAILhttps://phabricator.wikimedia.org/T173269EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: WMDE-leszek, jcrespo, Peachey88, Marostegui, Krenair
jcrespo added a comment.
@thcipriani @aaron I know something was done yesterday, (thank you!), may I ask for an update of the state, to know if the UBN is still active?TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
jcrespo closed this task as "Resolved".jcrespo claimed this task.jcrespo added a comment.
Resolving for now.TASK DETAILhttps://phabricator.wikimedia.org/T173269EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: WMDE-leszek, jcrespo,
jcrespo reassigned this task from jcrespo to daniel.
TASK DETAILhttps://phabricator.wikimedia.org/T173269EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, jcrespoCc: WMDE-leszek, jcrespo, Peachey88, Marostegui, Krenair, Addshore, TerraCodes, Jay8g
jcrespo added a project: CirrusSearch.Herald added projects: Discovery, Discovery-Search.
TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi
jcrespo added a comment.
So this is deployed into production, we did a test run and it seems to work as intended.
I left a "disable" patch https://gerrit.wikimedia.org/r/373507 and instructions to deploy there, in case fellow ops have to do some emergency thing to disable, so it is alr
jcrespo added a comment.
This is probably a symptom and not a cause, but I wanted to comment it anyway in case it was interesting:
There seems to be higher than usual hhvm exceptions:
https://logstash.wikimedia.org/goto/80fa5708f0a5e9da4be9f4630969b72e
Most of those, at least the ones that are
jcrespo added a comment.
@Ladsgroup: This should be easy to fix:
root@terbium:/var/log/wikidata$ ls -lha rebuildTermSql*
-rw-rw-r-- 1 www-data www-data 1.7K Aug 25 06:30 rebuildTermSqlIndex.log
-rw-r--r-- 1 www-data www-data 130 Aug 8 13:15 rebuildTermSqlIndex.log-20170810.gz
-rw-rw-r-- 1 www
jcrespo added a comment.
Of course, that doesn't apply to cases that are limited by a common resource (e.g. database).
If I could add to the ideal scenario, the jobqueue would have dedicated slaves AND would write with a different domain id (allowing parallelism) than the rest of the writes
jcrespo added a comment.
Could, at least, that part have something to do with T164173, as a problem from the same cause, or a consequence of the fix? I also remember some tunning of some wikidata crons or job size, but not sure if upwards or downwards and not sure if related. Aaron, Ladsgroup or
jcrespo added a comment.
This is my view of the issue, based on the comments above.
Short term:
If the number of watchlist items for the users is less than N (N to be determined), join Watchlist -> recentchanges, then "suffer" a small in-memory sort (I belive this is the current si
jcrespo added a comment.
Using STRAIGHT_JOIN and making sure an index is used more efficiently could help too (here I change the original query to order by rc_timestamp, and the explain is better, but the query is actually slower P5988#33002).TASK DETAILhttps://phabricator.wikimedia.org
jcrespo added a comment.
I am testing with a new index on dbstore1002; meanwhile I had a chat with Bawolff and he mentioned that rc used to be a small table where many indexes an inefficient scanning was possible because it was a much smaller summary of revision. Apparently with the latest
jcrespo added a comment.
After testing some indexes, I do not see a huge improvement- we can reduce from scanning 100M rows to 18M, but there can be always a combination of query parameters that does not filter many rows on recentchanges. Paging by id (or timestamp) is the only reliable solution
jcrespo added a comment.
We identify and delete duplicate rows (not trivial, but not difficult either), then we add a UNIQUE restriction over that combination of columns so that never happens again.TASK DETAILhttps://phabricator.wikimedia.org/T163551EMAIL PREFERENCEShttps
jcrespo added a comment.
Not sure if with "you", you mean me, but if it is safe, yes. We may have to defragment the table later to reclaim disk space, but that can be done later and it is not a blocker.TASK DETAILhttps://phabricator.wikimedia.org/T163551EMAIL PREFER
jcrespo added a comment.
@hoo Regarding Wikimedia setup, you must know that it is our priority right now to move wikidata to a dedicated server group; which means from ops side no other structural change can happen at the same time.
This is still needed (and doing it is blocked on code being
jcrespo added a comment.
I know- it is only related because the wikidata migration require replication channels movement and that consumes DBA time, not because it contains wikidata.TASK DETAILhttps://phabricator.wikimedia.org/T176273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings
jcrespo added a comment.
Cool, get if you can some `SHOW TABLE STATUS like stats, to get the "before" state in bytes (even if it had started some time ago).TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpref
jcrespo added a comment.
@Catrope Unfortunately it will take quite a bit of time to research and investigate the index, and I'm not able to attend to this right now due to several other currently happening infrastructure fires going on. I already commented some possibilities and I am open to
jcrespo added a project: DBA.
TASK DETAILhttps://phabricator.wikimedia.org/T174044EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Aklapper, daniel, E1presidente, Ramsey-WMF, Jmmuguerza, SandraF_WMF, GoranSMilovanovic, QZanden, Acer
jcrespo removed a project: Blocked-on-schema-change.jcrespo added a comment.
No actionables here (yet), add #blocked-on-schema-change when it actually is blocked on that.TASK DETAILhttps://phabricator.wikimedia.org/T174044EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
jcrespo added a comment.
@Anomie is right, adding it now would only spam us and deplay actually blocked changes. Add the DBA to tell us we will be involved in the future, as a heads up, which I already did (the Blocked external is the meaning here, not the not db team, we *will* be involved, just
jcrespo added a comment.
I am a bit lost with the estimation- is that realistic, is the number of usages more or less right with your estimation, or is it going to be a quadratic _expression_ because we do not know accurately the number of statement usages per statement? If the estimations are
jcrespo removed a project: Blocked-on-schema-change.jcrespo moved this task from Backlog to Blocked external/Not db team on the DBA board.jcrespo added a comment.
Add #blocked-on-schema-change, that way we will know it is ready :-)TASK DETAILhttps://phabricator.wikimedia.org/T177601WORKBOARDhttps
jcrespo raised the priority of this task from "High" to "Unbreak Now!".jcrespo claimed this task.jcrespo added a comment.
The introduction of wikidata events on recentchanges has converted the "light" recentchanges table into a monolithical 500GB table:
common
301 - 400 of 588 matches
Mail list logo
