jcrespo added a comment.
One tip to avoid having people on call (like me) worrying about pending
implementation services is to add the hiera key
`profile::monitoring::notifications_enabled: false`. This is not promoted much
because most people handle stateless services that are easy
jcrespo added a comment.
I was told by @Gehel that it was unrelated to this, but related to T301167
<https://phabricator.wikimedia.org/T301167>. Sorry for the confussion.
TASK DETAIL
https://phabricator.wikimedia.org/T323096
EMAIL PREFERENCES
https://phabricator.wikimedia.org/se
jcrespo added a comment.
Sorry if it is the wrong ticket, but several services of wdqs2010, wdqs2011
and wdqs2012 are alerting. The sevice is returnin 400 commands. My guess is
this is due to this ongoing data reload (no issue). If that is the case, could
the alerts "WDQS SPARQL"
jcrespo added a comment.
In T307586#7908045 <https://phabricator.wikimedia.org/T307586#7908045>,
@Quiddity wrote:
> For Tech News purposes, how should this entry be described? IIUC from the
description, something like this?
>
>> There was a problem with
jcrespo added a comment.
Thanks for such a quick reaction, BTW.
TASK DETAIL
https://phabricator.wikimedia.org/T307586
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: brennen, hashar, jcrespo, Raymond, Moebeus, Lucas_Werkmeister_WMDE
jcrespo updated the task description.
jcrespo added a project: User-notice.
TASK DETAIL
https://phabricator.wikimedia.org/T307586
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, Raymond, Moebeus, Lucas_Werkmeister_WMDE, Aklapper
jcrespo added a comment.
In addition to the fix/rollback- could some integration test or heuristic
production monitoring also be able to be implemented, for future faster
detection?
TASK DETAIL
https://phabricator.wikimedia.org/T307586
EMAIL PREFERENCES
https
jcrespo added a comment.
Terminated Jobs:
JobId Level FilesBytes Status FinishedName
396417 Full 108,32011.70 G OK 13-Dec-21 09:34
graphite1004.eqiad.wmnet-Weekly-Mon
jcrespo added a comment.
Running Jobs:
Console connected using TLS at 13-Dec-21 09:20
JobId Type Level Files Bytes Name Status
==
396417 Back Full 4,568412.9 M
graphite1004
jcrespo added a comment.
Let me give it a deeper look, while the patch by itself looks good as is, I
want to check if a different (non-default) backup policy would be more
advantageous in frequency and space. :-)
TASK DETAIL
https://phabricator.wikimedia.org/T294355
EMAIL PREFERENCES
jcrespo added a comment.
I don't have the answer to that question, but whenever any of you have the
servers and path(s), you can follow the instructions at
https://wikitech.wikimedia.org/wiki/Bacula#Adding_a_new_client to send a
preliminary backup proposal to Puppet, and I will assist you
jcrespo added a comment.
One more question, to finally decide if setting up weekly full backups or
daily but incremental- do all files mostly change completely, or only a subset
of them? Incrementals are able to be done with file granularity only (it will
backup fully files as long as its
jcrespo added projects: bacula, Data-Persistence-Backup, Data-Persistence.
jcrespo added a comment.
number of files are (within reason) a non-blocker for bacula, as files are
packaged into volumes. It is true that each file is stored as a mysql record,
but that should be able to scale until
jcrespo added a project: Data-Persistence (Consultation).
TASK DETAIL
https://phabricator.wikimedia.org/T186716
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Kormat, Michael, Ladsgroup, Marostegui, RP88, Mike_Peel, Aklapper
jcrespo added a subscriber: Kormat.
jcrespo added a comment.
> the DBAs to approve this
That should be @Kormat and/or @Marostegui (he is on vacations right now).
TASK DETAIL
https://phabricator.wikimedia.org/T186716
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/pa
jcrespo added a comment.
At 16:02-16:06, which would fit with the deployment of
https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/725019/ logstash
baseline number of messages increased around a 50%:
https://grafana.wikimedia.org/d/00561/logstash?orgId=1=2=1633013879325
jcrespo triaged this task as "Unbreak Now!" priority.
jcrespo added a comment.
This should be a blocker- es traffic has grown almost grown 100x since 14
april, correlates strongly with the 19h deploy:
F34434387: es_issue.png <https://phabricator.wikimedia.org/F34434387&
jcrespo added a comment.
Offtopic- and feel free to PM in private. What is a good way to report
database-related issues to wikidata development team? I am a bit intimidated by
the amount of tags and dashboards (which will probably reflect your internal
organization, but I am not too
jcrespo updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T276762
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Marostegui, hoo, Aklapper, jcrespo, maantietaja, Akuckartz, darthmon_wmde,
Nandana, Lahi, Gq86
jcrespo created this task.
jcrespo added projects: Wikidata, Wikibase.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
This weekend, while an ongoing incident was being handled, I checked and saw
several badly performing queries running. These didn't have (I believe) any
jcrespo added a subscriber: Marostegui.
jcrespo added a comment.
Handover to @Marostegui for him to comment, as he will be the person to know
if this continues happening or now.
TASK DETAIL
https://phabricator.wikimedia.org/T138208
EMAIL PREFERENCES
https://phabricator.wikimedia.org
jcrespo created this task.
jcrespo added projects: Wikidata, MediaWiki-extensions-PropertySuggester,
Wikimedia-production-error.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
We got an alert on `#wikimedia-databases` IRC saying:
PROBLEM - MariaDB sustained
jcrespo closed subtask T165756: Create summary templates on Wikitech wiki to
stop writing the same things everywhere, everytime as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T165726
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo added a comment.
To be fair, technically, this is resolved because wb_terms has been replaced,
AFAIK, with a more specialized mechanism (several smaller and normalised
tables) :-)
TASK DETAIL
https://phabricator.wikimedia.org/T86530
EMAIL PREFERENCES
https
jcrespo closed subtask T172490: Monitor swap/memory usage on databases as
Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T171928
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: gerritbot, mark, Marostegui, Elitre, Joe, jcrespo
jcrespo added a comment.
There is this procedure:
https://wikitech.wikimedia.org/wiki/MariaDB/query_performance which could be
done for Wikidata by people with root database privileges, however it is quite
involved. I will ask if maybe Manuel wants to perform soon such a thing for
other
jcrespo added a comment.
> based on that sentence, I would even today vote against the change Amir
just uploaded if it wasn’t for this ticket, because it seems to fly directly in
the face of this documentation: why use $join_conds if the documentation says
it’s unnecessary except for a L
jcrespo added a comment.
One thing that may be relevant here is that the query may have worked once or
twice in the last 6 months due to this underlying issue. Last update that
completed successfully was 27 March. In effect, because this bug, this wasn't
disabled officially
jcrespo added a comment.
{P11212}
TASK DETAIL
https://phabricator.wikimedia.org/T252952
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Addshore, jcrespo
Cc: WMDE-leszek, darthmon_wmde, Lea_Lacroix_WMDE, Marostegui, jcrespo,
Ladsgroup
jcrespo added a project: Wikimedia-Incident.
TASK DETAIL
https://phabricator.wikimedia.org/T238199
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Lea_Lacroix_WMDE, jcrespo, Addshore, Lydia_Pintscher, Aklapper, Ladsgroup,
Marostegui
jcrespo added a comment.
Over 50K user errors:
https://logstash.wikimedia.org/goto/e78777bdaa039396e05bca8099294c91
TASK DETAIL
https://phabricator.wikimedia.org/T238199
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc
jcrespo closed subtask T172489: Monitor read_only on all databases, make it
page on masters as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T171928
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: gerritbot, mark, Marostegui
jcrespo changed the status of subtask T172489: Monitor read_only on all
databases, make it page on masters from Stalled to Open.
TASK DETAIL
https://phabricator.wikimedia.org/T171928
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc
jcrespo changed the status of subtask T239238: Switchover s8 primary database
master db1109 - db1104 - Date TBD from Open to
Stalled.
TASK DETAIL
https://phabricator.wikimedia.org/T237120
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Marostegui
jcrespo added a comment.
I think people that worked for a long time since yesterday are taking a
well-deserved rest and may be unresponsive. From the data recovery/database
perspective everything is done and no longer UBN. But I think it should be
someone familiar with Wikidata that should
jcrespo added a comment.
> perhaps it might be an idea for more people to have access to this zarcillo
thing?
Stress on "The reasons why those are not more public is that we were told not
to put those on public prometheus by security as they could compromise the
anonymity of
jcrespo added a comment.
Looks like CPU exhaustion which could be, in part, due to T232446
<https://phabricator.wikimedia.org/T232446>.
TASK DETAIL
https://phabricator.wikimedia.org/T246159
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailprefe
jcrespo added a comment.
T238199 <https://phabricator.wikimedia.org/T238199> is not a duplicate, but
certainly it is the same theme and effect (just caused lag on wikidata
replicas).
TASK DETAIL
https://phabricator.wikimedia.org/T235265
EMAIL PREFERENCES
jcrespo edited projects, added MediaWiki-Special-pages; removed
MediaWiki-General.
TASK DETAIL
https://phabricator.wikimedia.org/T238199
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, Addshore, Lydia_Pintscher, Aklapper
jcrespo added a project: Wikimedia-production-error.
jcrespo added a comment.
https://logstash.wikimedia.org/goto/060d35b0c4c75be0a47e14bb8602ac14
TASK DETAIL
https://phabricator.wikimedia.org/T238199
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo added a comment.
I killed SpecialFewestRevisions::reallyDoQuery because it was causing the
host (s8, db1101) where it was running to lag for 1-2 seconds, enough for
queries waiting for replication to take a dive in performance, and logging
hundreds of thousands of db errors. Long
jcrespo updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T246232
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Aklapper, Marostegui, Ladsgroup, jcrespo, darthmon_wmde, Nandana, Lahi,
Gq86, GoranSMilovanovic
jcrespo created this task.
jcrespo added a project: Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
At T246159 <https://phabricator.wikimedia.org/T246159> we can observe a
really badly formatted query:
SELECT /*
Wikibase\Lib\Store\Sql
jcrespo updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T246232
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Aklapper, Marostegui, Ladsgroup, jcrespo, darthmon_wmde, Nandana, Lahi,
Gq86, GoranSMilovanovic
jcrespo added a comment.
so 200K per minute overall is approximately 20/60/3 (db1126 has a third
of the main, non special traffic) = 1000 QPS max per server. I would categorize
this as frequent, normally those should have a latency of 0.01 seconds or less
in average
jcrespo added a comment.
I am almost made up my mind to give a suggestion, last question, I promise:
could you provide an idea of frequency of this query (normally & at peak
times). This will be the determinant factor on how much latency we can "spend"
on it. I don't need exac
jcrespo added a comment.
Do you have latency data for each run? Normally clients report query
execution time, which you should already have when getting the handlers.
TASK DETAIL
https://phabricator.wikimedia.org/T246159
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
jcrespo added a comment.
Thanks a lot for the work on this- may I suggest a step before the next step
(after rebuilding) of "checking all data, old and new, is consistent". This is
a lot of data, and even on well thought processes missing rows were discovered
after I requested a
jcrespo added a comment.
> API maxlag has been repeatedly declared to always be <5s by several people
in the past
That would sound about right for Mediawiki API (5-10 seconds) ...
> If WDQS lag is high, API maxlag is also high
... but this sounds worrying if true- I w
jcrespo added a comment.
I am not part of the Wikidata QS team, so I don't have answers, just
questions :-D Only chiming in because my team was been tagged on this ticket-
please understand we (SREs) are not in direct charge of this service and that
someone else should answer with first
jcrespo edited projects, added Core Platform Team; removed Core Platform Team
Legacy (Watching / External).
jcrespo added a comment.
This is flapping very frequently, but with a 500, not a 429 (scb1002 only,
for example, twice per hour). Should I close this and open a new one, or can
jcrespo added a comment.
This is solved to me, thanks @Gehel
https://grafana.wikimedia.org/d/00489/wikidata-query-service?orgId=1=8=1577192533117=1577210803000
TASK DETAIL
https://phabricator.wikimedia.org/T241418
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
jcrespo added a comment.
Issue started the 22 Dec at around 2:16
https://grafana.wikimedia.org/d/00377/host-overview?orgId=1=wdqs1006=eqiad%20prometheus%2Fops=wdqs=1576541970193=1577197748564
TASK DETAIL
https://phabricator.wikimedia.org/T241418
EMAIL PREFERENCES
https
jcrespo added a comment.
Issue started the 22 Dec at around 2:16
https://grafana.wikimedia.org/d/00377/host-overview?orgId=1=wdqs1006=eqiad%20prometheus%2Fops=wdqs=1576541970193=1577197748564
TASK DETAIL
https://phabricator.wikimedia.org/T213191
EMAIL PREFERENCES
https
jcrespo created this task.
jcrespo added projects: Wikidata-Query-Service, Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
24 Dec 13:38 PROBLEM - Disk space on wdqs1006 is CRITICAL: DISK
CRITICAL - free space: /srv 53349 MB (3% inode=99%)
TASK DETAIL
https
jcrespo added a comment.
I have provided @addshore with the transaction that deleted the `wbx_id =
110010309` row. It happened on 2019-12-05 11:09:27 (although the transaction,
and its data context, started in the previous second).
TASK DETAIL
https://phabricator.wikimedia.org/T237984
jcrespo created this task.
jcrespo added a project: Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
While this error is just a warning (not fatal), please triage to see if it
breaks some intended logic:
{
"_index": "logstash-de
jcrespo added a comment.
Unless there is a last time change, this issue is for me fixed. Obviously,
please do any data checks/refill that you think is necessary (if any) to fix
potential problems caused by this. Thanks for working on this.
TASK DETAIL
https://phabricator.wikimedia.org
jcrespo assigned this task to Marostegui.
TASK DETAIL
https://phabricator.wikimedia.org/T237120
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Marostegui, jcrespo
Cc: Marostegui, jcrespo, alaa_wmde, Addshore, Aklapper, Ladsgroup, Iflorez
jcrespo triaged this task as "Normal" priority.
jcrespo moved this task from Triage to Next on the DBA board.
jcrespo added a comment.
Thanks, acking this, but I hope this is not an emergency, as we may take a
bit more than usual to apply it, as 50% of persistence team is on unavai
jcrespo added a comment.
FYI There seems to be a 50% or more write penalty throughput when the rebuild
script is running:
https://grafana.wikimedia.org/d/00273/mysql?var-server=db1109=9104=eqiad%20prometheus%2Fops=1=1572439082952=1572460682953
TASK DETAIL
https
jcrespo added a comment.
> The maintenance script restarted every 30 mins
Small correction, every hour at XX:30 minutes. Maybe you already knew that,
but just in case it could lead to missleading analysis.
TASK DETAIL
https://phabricator.wikimedia.org/T236928
EMAIL PREFEREN
jcrespo added a comment.
> We are seeing 5-15s lag on db slaves.
Offtopic- I don't believe the lag was real- it just that there was so much
overload that lag checking reads/connections by mw would take so much time to
finish and return results. This is irrelevant for this tic
jcrespo added a parent task: T30599: Deadlock tracking bug (tracking).
TASK DETAIL
https://phabricator.wikimedia.org/T236464
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, Liuxinyu970226, #dba, Aklapper, LarsWirzenius
jcrespo added a comment.
Is this on Wikidata? T234948 <https://phabricator.wikimedia.org/T234948>
TASK DETAIL
https://phabricator.wikimedia.org/T236464
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, Liuxinyu970226
jcrespo added a comment.
We found a possible cause even if partial: maintenance queries for Special
page update like `SpecialMostLinked::reallyDoQuery` causing lag and general
slowdown on other servers, probably you can check what those do and if they
break or make the wikidata migration
jcrespo added a comment.
Now there seems to be also many duplicate insert errors.
TASK DETAIL
https://phabricator.wikimedia.org/T234948
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: alaa_wmde, jcrespo
Cc: alaa_wmde, Jdforrester-WMF, WMDE-leszek
jcrespo added a comment.
It came back
https://logstash.wikimedia.org/goto/de0f6611dde37e811edcba4f530131c0
TASK DETAIL
https://phabricator.wikimedia.org/T234948
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: alaa_wmde, jcrespo
Cc: alaa_wmde
jcrespo added a comment.
> @jcrespo what rate should we be concerned about? (to plan/monitor ahead)
As I said above, with the current rate it is not of immediate concern to me,
but my suggestion is to understand why it is happening and what are the
consequences, to be 100% sure d
jcrespo added a parent task: T30599: Deadlock tracking bug (tracking).
TASK DETAIL
https://phabricator.wikimedia.org/T234948
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: WMDE-leszek, Marostegui, Ladsgroup, Aklapper, jcrespo
jcrespo created this task.
jcrespo added projects: Wikidata, Wikimedia-database-error,
Wikimedia-production-error.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Before 2019-10-08T00:00:02 there was almost no wikidata deadlocks, since then
there are 650-1250 per hours
jcrespo added a comment.
We already have sizes of all uncompressed and compressed tables on zarcillo,
those are planned to be shown in a dashboard. The reasons why those are not
more public is that we were told not to put those on public prometheus by
security as they could compromise
jcrespo added a comment.
Yes, we were worried about the frequency of it (it was the most common
DB-related error for some time) because of the performance implications, not
that it happens sometimes. I am guessing that this is due to multiple edits
happening at the same time about the same
jcrespo added a comment.
I am going away now for this week, but if I happen to be right about my
previous comment, please don't close the ticket, and make this a child of
T220735 <https://phabricator.wikimedia.org/T220735>. If the issues is sill
persisting, it is something else
jcrespo added a comment.
> since yesterday
Could you try again? There was a regression on .10 that now has been
reverted, and the timing matches the .10 deployment to wikidata. This is still
a bug if it no longer happens, and we need to notify so it doesn't reappear (we
are now on
jcrespo added a comment.
Apparently there are jobqueue issues, potentially impacting to this: T226109
<https://phabricator.wikimedia.org/T226109>
TASK DETAIL
https://phabricator.wikimedia.org/T205045
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailprefe
jcrespo added a comment.
Thanks, that looks good. I saw no significant impact impact on db infra, if
it does more queries/transactions it had not noticeable issue. Did it impact
lag/performance on your side?
I see a gap here last week-
https://logstash.wikimedia.org/goto
jcrespo added a comment.
Could you add the request url/post data used at T226084#5268060
<https://phabricator.wikimedia.org/T226084#5268060> ? Having 1 or more specific
endpoints simplifies debugging.
TASK DETAIL
https://phabricator.wikimedia.org/T226084
EMAIL PREFERENCES
jcrespo added a comment.
So just to be clear- based on growth projections I recently got from SDC team
of wikibase on commons, the separation is not only convenient or needed for
performance, literally s4 would not be able to fit except the initial
deployment of data, or do it for very
jcrespo added a comment.
> The move would likely require some changes to the Wikibase code
Could you clarify why? As all other hosts seem to be ok with wikibase server
for wikidata being on a separate database? Does MCR use wikibase differently or
is it something else? Maybe havin
jcrespo added a comment.
https://logstash.wikimedia.org/goto/fa3d82b03d920e028b05682a499c9f8b Thanks
a lot to everybody!
TASK DETAIL
https://phabricator.wikimedia.org/T221577
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: aaron, jcrespo
jcrespo added a comment.
Just to be clear, this is on HEAD, but not yet deployed/live, correct?
https://logstash.wikimedia.org/goto/9eb41e92236727f9095b28489dc244a6
No problem if I am right, just to make sure nothing is being missed.
TASK DETAIL
https://phabricator.wikimedia.org
jcrespo added a comment.
> and DBAs to babysit the deployment for a couple of days
I can do that if you tell me anyting specific -metrics- to check for (other
than typical error count and general stability. Specially if initial deployment
happens at the beginning of next w
jcrespo added a comment.
Because this issue, or T143870 <https://phabricator.wikimedia.org/T143870>,
and/or long running connections due to mw connection handler, there was a
connection issue at
https://logstash.wikimedia.org/goto/286304e84262d2fe3335acd5eed135bb and there
is
jcrespo added a comment.
FYI, because in the past I have been told by people they were not aware of
the impact. I am **not** too worried about this because I don't think it
creates user impact (it impacts only the jobqueue), but this is probably at the
moment the most frequent Wikibase
jcrespo added subscribers: Marostegui, jcrespo.
jcrespo added a comment.
I've been told wikibase tables have been created on s4. We would like to have
been notified of this- we are not sure wikibase for commons should live on s4,
if the growth of structured data is as large
jcrespo added a comment.
@Yann That is known, it has been on purpose left like that so an actual fix
can be tested there first.
TASK DETAIL
https://phabricator.wikimedia.org/T219450
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc
jcrespo added projects: Wikimedia-Incident, Wikimedia-production-error.
TASK DETAIL
https://phabricator.wikimedia.org/T219450
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, Jdforrester-WMF, TerraCodes, Liuxinyu970226
jcrespo added a comment.
We have several confirmations this is working now.
TASK DETAIL
https://phabricator.wikimedia.org/T219450
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, Jdforrester-WMF, TerraCodes, Liuxinyu970226
jcrespo added subscribers: Gehel, Joe, jcrespo.
jcrespo added a comment.
I see you are going to ask DBAs at T219145
<https://phabricator.wikimedia.org/T219145>, that is great.
As a heads up, because I saw there is a chance of starting using other data
stores (which by
jcrespo added a comment.
labsdb1009.mgmt (stress on management interface) is down according to icinga
for 14 hours (around net maintenance), maybe a loose cable or misconfiguration?
Not a huge blocker, but better make sure it is not an intended/known state.
TASK DETAIL
https
jcrespo added a subtask: T188679: Nuke should batch the deletions.
TASK DETAIL
https://phabricator.wikimedia.org/T212690
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, gerritbot, Addshore, Marostegui, Aklapper, abian, alaa_wmde
jcrespo added a comment.
I am going to be bold and say that maybe the solution to timeouts (or at
least part of the solution for timeouts) like this one reported on enwiki:
"PHP fatal error: entire web request took longer than 60 seconds and timed
out"
would be T188
jcrespo added a comment.
the API works good for query specific pages/entities, not for example to know which pages that existing in X_wiki are missing on the Y_wiki.
Sure, and you are free to either:
a) Setup your own database from dumps or even expose it to other people
b) Do a feature request
jcrespo added a comment.
diego added a project: DBA.
I don't understand what is the actionable here for us. Without context, I would say that:
(1) querying the wb_items_per_site table in the wikidatawiki on MariaDB, or
(2) through the sitelinks on Wikidata Json dumps.
That is not accurate
jcrespo added a subtask: T104459: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves.
TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc
jcrespo removed a parent task: T104459: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves.
TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo added a comment.
Thanks @Addshore !TASK DETAILhttps://phabricator.wikimedia.org/T214644EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Ladsgroup, Addshore, Aklapper, abian, Zeroth, Legado_Shulgin, Nandana, thifranc, AndyTan
jcrespo added a comment.
The edit rate not being affected means this was a probably relatively localized problem https://grafana.wikimedia.org/d/00208/edit-count?orgId=1=1548357720872=1548374190181 and not something that happens regularly.TASK DETAILhttps://phabricator.wikimedia.org
jcrespo added subscribers: Ladsgroup, jcrespo.jcrespo triaged this task as "Low" priority.jcrespo moved this task from Triage to Blocked external/Not db team on the DBA board.jcrespo added a comment.
Sorry you suffered this issues. I can see a few hundred Wikibase\UpsertSqlIdGenerator
1 - 100 of 581 matches
Mail list logo
