[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[ArielGlenn]

ArielGlenn added a comment.


  Retries of wikidata entity dumps rerun the particular batch from a MediaWiki 
maintenance script; it will set up configuration from scratch.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, darthmon_wmde, 
Premeditated, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[jcrespo]

jcrespo added a comment.


  Because this issue, or T143870 , 
and/or long running connections due to mw connection handler, there was a 
connection issue at 
https://logstash.wikimedia.org/goto/286304e84262d2fe3335acd5eed135bb and there 
is likely to be another one. This may or may not create isues on wikidata 
dumps/exports, depending if retries are done on the latest configuration.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, darthmon_wmde, 
Premeditated, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[hoo]

hoo added a comment.
One thing we could possibly do next: Add a hook in getConnection (or somewhere close) that let's us kill the connection attempt (or the entire script) in case an unwanted replica is selected.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, gnosygnu, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[hoo]

hoo added a comment.

In T138208#4679458, @jcrespo wrote:
From another thread, from TimS:

It sounds like the snapshot hosts were in fact needing and using the host. It's not LoadBalancer's fault if some maintenance script calls wfGetDB() without specifying a query group. Throwing an exception is the correct thing to do if the MW configuration is incorrect, since it allows the maintenance script to terminate and be restarted with different configuration.


Given we set $wgDBDefaultGroup* to dump for these scripts, LoadBalancer should only ever use the "default group" in case the dump host(s) are not available. Given this (probably) didn't happen here, we're almost certainly facing some other problem.

* $wgDBDefaultGroup configures the DB group to use per default (if no other group is explicitely given), thus wfGetDB (and others) will not connect to the default hosts.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, gnosygnu, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[jcrespo]

jcrespo added a comment.
From another thread, from TimS:

It sounds like the snapshot hosts were in fact needing and using the host. It's not LoadBalancer's fault if some maintenance script calls wfGetDB() without specifying a query group. Throwing an exception is the correct thing to do if the MW configuration is incorrect, since it allows the maintenance script to terminate and be restarted with different configuration.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, gnosygnu, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[ArielGlenn]

ArielGlenn added a comment.
@jcrespo if you do not indeed see this on any other shards, it's probably not anywhere in the code I write/run, which is why I ask if you've seen it elsewhere except wikidata.

Also, I'm happy to look at the Wikidata-specific code and help make sure that the right db is used for these jobs.

Beyod that I would like the ability to tell LB to drop the current connection AND config and re-read.  This would be very handy in general.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArielGlennCc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[Lydia_Pintscher]

Lydia_Pintscher added a comment.
@jcrespo How much fire is this? Does it need fixing while we are at Wikimania or is ot ok if we handle it after that?TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_PintscherCc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[jcrespo]

jcrespo added a comment.
However, there are potentially several dozen places where we call LoadBalancer::getConnection (we try not to hog the connection, but only get it from the LB when we need it - so we do that often). We'd have to somehow loop this parameter through to all the places where we get the connection. Or is it sufficient to specify the group on the first call? Will the LB return a connection that was created for a specific group, if later no group is specified in the call to getConnection?

I cannot say by heart, I will have to check. We may want to loop in performance team as they gave good tips last time we had an issue with connection handling and tumbnails. I really do not think this is a general problem in all of wikidata, just in a couple of scripts- some scripts keeps connected for 9 minutes, mostly idle- that is the main issue.

However, for what you say, you may want to refactor the connection handling (wikidata's, not mediawiki's) to a more abstract-factoy-likey, as role can be 'dump' now and 'wikidata-special-dump' tomorrow (that is not actually happening, but it could).TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: daniel, hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[daniel]

daniel added a comment.
@jcrespo: is this triggered only by dispatchChanges.php, or also by dumpJson.php or dumpRdf.php?TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: daniel, hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[daniel]

daniel added a comment.
@jcrespo

These changes literally only need one parameter change loadbalancer->get(SLAVE, 'dump')

That sounds straight forward enough.

However, there are potentially several dozen places where we call LoadBalancer::getConnection (we try not to hog the connection, but only get it from the LB when we need it - so we do that often). We'd have to somehow loop this parameter through to all the places where we get the connection. Or is it sufficient to specify the group on the first call? Will the LB return a connection that was created for a specific group, if later no group is specified in the call to getConnection?TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: daniel, hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[jcrespo]

jcrespo added a comment.
Let me show you the weight of these servers:

	's5' => array(
		'vslow' => array(
			'db1045' => 1,
		),
		'dump' => array(
			'db1045' => 1,
		),
		'api' => array(
			'db1070' => 1,
			'db1071' => 1,
		),
		'watchlist' => array(
			'db1026' => 1,
		),
		'recentchanges' => array(
			'db1026' => 1,
		),
		'recentchangeslinked' => array(
			'db1026' => 1,
		),
		'contributions' => array(
			'db1026' => 1,
		),
		'logpager' => array(
			'db1026' => 1,
		),
	),

Do you see db1071 anywhere? Not on dump, not on vslow (and I just added it to api, it was not there before, when the issue started happening). So yes, I only expect most short-lived connections (the ones created by non-api end-user requests) to only go to the new servers while they are being tested. This separation is important due to performance and HA reasons. By not following mediawiki standards you are threatening the reliability of the site, for wikidatawiki and for dewiki users (which are already not happy).

There is not need to change any mediawiki architecture. terbium jobs must use the 'vslow' role. If you think only one server is not enough, please say so and we change the config to add more servers to vslow (which we certainly can do now that we have more servers), but not violate mediawiki contract of roles sending long-running connections to servers that are not ready for them.

The same thing applies for dumps- if more are needed, configuration is changed, not hardcoding the wrong ones.

These changes literally only need on parameter change loadbalancer->get(SLAVE, 'dump')TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: daniel, hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[daniel]

daniel added a comment.
My 2¢

@jcrespo wrote

If these create light-weight queries only, lets disconnect and connect after some amount of seconds.

Teaching LoadBalance to throw away connections after a minute or so would be easy. But would that be sufficient? Reconnecting would use the old configuration that was valid when the process started. Reconfiguring on the fly is a lot more tricky.

If dump hosts are too slow, let's give them better resources.

Yes, please. When dump scripts run for weeks instead of days, they tend to fail at some point. I agree that we shouldn't hit the web-facing databases for making dumps. But if that means we get no dumps at all, that's not good either...

If there is a problem with the connection framework, let'x fix it with the addition of a proxy/persistent connections manager.

My impression is that what we want here is the opposite of persistent connections. Making it possible to re-configure MediaWiki mid-air would certainly be nice, it's something I have been working on with respect to dependency injection and unit testing.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: daniel, hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[hoo]

hoo added a comment.

In T138208#2398369, @jcrespo wrote:
As I said:

If these create light-weight queries only, lets disconnect and connect after some amount of seconds. If dump hosts are too slow, let's give them better resources. If there is a problem with the connection framework, let'x fix it with the addition of a proxy/persistent connections manager.

By using the wrong hosts you are creating connection issues for end users.


Why do testing slaves have weight? If they have weight, you need to expect MediaWiki to connect to them. As said, we can (rather easily) reduce the run-time of the terbium scripts. Regarding the dumps: I'm in a conversation with Daniel about how to fix this… but it's not easy given that it requires a significant change in MediaWiki (also architecture wise).TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[jcrespo]

jcrespo added a comment.
As I said:

If these create light-weight queries only, lets disconnect and connect after some amount of seconds. If dump hosts are too slow, let's give them better resources. If there is a problem with the connection framework, let'x fix it with the addition of a proxy/persistent connections manager.

By using the wrong hosts you are creating connection issues for end users.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[jcrespo]

jcrespo added a comment.
These jobs need access to the main DBs

What is a "main db" and what is the difference with a 'vslow' slave? You are accessing a testing slave, that will be put down at any moment (or will block & kill terbium traffic).TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[hoo]

hoo added a comment.
Regarding the dump scripts running on the snapshot hosts: If needed, I can try to make these use the "dump" servers, although that's going to significantly slow them down (given we use queries that the API and the UI also uses). Reloading the database configuration is sadly not going to be easy, given we have these definitions in PHP.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[hoo]

hoo added a comment.

In T138208#2393678, @jcrespo wrote:
This is not only happening for dumps, terbium is also wrongly using main-dbs (which are still on testing) for long-running queries, which cause long periods of connection issues: https://logstash.wikimedia.org/#dashboard/temp/AVVuJmxb_LTxu7wlh30V

root@terbium:~$ lsof | grep 43894
php5  32192  www-data8u IPv4 1620677566   0t0TCP terbium.eqiad.wmnet:43894->db1092.eqiad.wmnet:mysql (ESTABLISHED)
root@terbium:~$ ps aux | grep 32192
root 16294  0.0  0.0  11864   916 pts/5S+   14:06   0:00 grep 32192
www-data 32192  6.3  0.1 333200 53700 ?S13:57   0:34 php5 /srv/mediawiki-staging/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dispatchChanges.php --wiki wikidatawiki --max-time 540 --batch-size 275 --dispatch-interval 25 --lock-grace-interval 200
root@terbium:~$ lsof | grep 52769
php5   9169  www-data8u IPv4 1620971979   0t0TCP terbium.eqiad.wmnet:52769->db1092.eqiad.wmnet:mysql (ESTABLISHED)
root@terbium:~$ ps aux | grep 9169
www-data  9169  7.5  0.1 330380 50984 ?S14:03   0:16 php5 /srv/mediawiki-staging/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dispatchChanges.php --wiki wikidatawiki --max-time 540 --batch-size 275 --dispatch-interval 25 --lock-grace-interval 200
root 22859  0.0  0.0  11868   916 pts/5S+   14:06   0:00 grep 9169


These jobs need access to the main DBs as well as to the master (these jobs are used to actively dispatch incoming changes to various Wikipedias). Currently we let these scripts run for about 9 minutes… if needed, we can cut that down (they run on Zend, so the performance penalty of restarts is rather low).TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[jcrespo]

jcrespo added a comment.
10.64.48.26 is db1071, a regular-traffic db.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[jcrespo]

jcrespo added a comment.
Right now my only worries are for wikidata, because they create large amount of connection errors and are very visible- I can check on other shards, but even if they do, it would be very low priority (no infrastructure issues there).TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[ArielGlenn]

ArielGlenn added a comment.
Clarification: is this *only* happening for wikidata, or do you notice this for other wiki dumps too?TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArielGlennCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[jcrespo]

jcrespo added a comment.
@ArielGlenn : let's identify the reason why this is happening before changing things- we may implement proxying/etcd config before changing any logic. The important part here and now is that probably some mediawiki class is not using the dump role, maybe related to wikidata?TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

[ArielGlenn]

ArielGlenn added a comment.
I sure expect the dump db hosts to be used; if they are not be, that's a bug somewhere in the MW code.  This used to work properly AFAIR.

In the meantime closing and reopening connections after some number of seconds or some small number of requests is fine.  I can look into that.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArielGlennCc: ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs