Re: [Wikimedia-l] [Cloud] [Cloud-announce] New Wikireplicas available, timeline update, and Quarry migration

2021-03-15 Thread Joaquin Oltra Hernandez 
Hi,

These changes are not arbitrary, they are a necessity to keep the replicas
functioning. With the growth experienced in the last years (commons,
wikidata) the current technology used by Wikireplicas is not feasible. I’ve
outlined more technical details in a cloud mailing list response: [1]

We have been keeping track of all the code posted that uses cross-wiki
joins, and are planning to analyze more from logs. [2] was made to figure
out common use cases and possibilities to enable them.

If you need help evaluating how you will be affected or migrating your code
please reach out in the #wikimedia-cloud IRC channel,
cl...@lists.wikimedia.org mailing list or make a task in Phabricator
#Data-Services [3].

[1] https://lists.wikimedia.org/pipermail/cloud/2021-March/001436.html
[2] https://phabricator.wikimedia.org/T215858
[3] https://phabricator.wikimedia.org/project/board/2874/

On Sun, Mar 14, 2021 at 12:15 AM Daniel Schwen  wrote:

> This is the next step after disallowing user databases on replicas. It
> broke some of my tools but I recently rewrote them to move joining logic
> into my application. I also replicate small amounts of data (e.g. page
> titles for a subset of pages) into my user db for joins.
> I found it quite off-putting at first. Volunteers are effectively forced
> to do substantial amounts of extra work after creating tools that "work
> perfectly fine".
> But I can understand the need to provide more scalable infrastructure with
> the ever growing projects.
>
> On Sat, Mar 13, 2021, 3:46 PM Yetkin Sakal via Cloud <
> cl...@lists.wikimedia.org> wrote:
>
>> I completely agree with Maarten. It would be a step backward to stop
>> supporting cross-database joins on wiki replicas. This is a breaking change
>> and should not be applied unless a feasible solution to the problem is
>> found.
>> On Saturday, March 13, 2021, 8:17:39 PM GMT+3, Maarten Dammers <
>> maar...@mdammers.nl> wrote:
>>
>>
>> Hi Joaquin,
>>
>> Completely nothing was done with the community input about not being able
>> to do cross wiki joins anymore [1]. In the past the WMF would do something
>> with community input. I guess the new strategy is to just give the
>> appearance of community input to legitimize a decision. Nice way to
>> alienate the volunteers.
>>
>> Maarten
>>
>> [1]
>> https://lists.wikimedia.org/pipermail/cloud/2020-November/thread.html#1309
>> On 12-03-2021 19:16, Joaquin Oltra Hernandez wrote:
>>
>> TLDR:
>> - Instead of `*.db.svc.eqiad.wmflabs` use `*.db.svc.wikimedia.cloud` to
>> use the new replicas
>> - Quarry will migrate March 23 to use the new cluster
>> - In a ~month (April 15) the old cluster will start retiring. See
>> https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeline
>> for more details
>>
>> Hi everyone,
>>
>> I'm happy to announce that the new replicas cluster is available for use
>> after a few weeks open for testing.
>>
>> To use the new cluster, you will only need to change the hostname when
>> connecting to the databases. Instead of `*.db.svc.eqiad.wmflabs` you can
>> use `*.db.svc.wikimedia.cloud`. See:
>>
>> -
>> https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_host_names
>> -
>> https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_can_I_test_the_new_replicas_before_the_switchover?
>> - https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database
>>
>>
>> This brings us to the timeline updates:
>>
>> https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeline
>>
>> - March 2021
>>   - Announce new cluster domains for all users - March 12
>>   - Migrate Quarry to use the new cluster - March 23
>> - April 2021
>>   - PAWS migration - Estimated first week of April
>>   - Migrate the old cluster to utilize new replication hosts. Replication
>> may stop. - April 15
>>   - Redirect old hostnames to the new cluster - April 28
>>
>> Quarry is ready so it will transition first to use the new replicas. This
>> will happen on March 23, at which point we will deploy the changes and
>> update the docs. We are publishing a note on Tech news but if there are
>> other venues where this information would be useful please help us spread
>> the word.
>>
>> PAWS is in the process of being migrated. We estimate it could be ready
>> to use the new cluster at the beginning of April, and will publish more
>> details as soon as we know.
>>
>> On April 15 the old cluster will be moved under the new replication
>> hosts, at which point there may be side effects and replication may stop.
>> We recommend you update your code earlier to use the new hostnames to
>> verify things are working normally.
>>
>> Not long after, the old hostnames (`*.db.svc.eqiad.wmflabs`) will be
>> redirected to the new ones (`*.db.svc.wikimedia.cloud`), at which point the
>> old cluster will effectively be inaccessible.
>>
>>
>> If you have questions or need help please reply to
>> cl...@lists.wikimedia.org, join the #wikimedia-cloud IRC channel, or

Re: [Wikimedia-l] [Cloud] [Cloud-announce] New Wikireplicas available, timeline update, and Quarry migration

2021-03-13 Thread Maarten Dammers 

Hi Joaquin,

Completely nothing was done with the community input about not being 
able to do cross wiki joins anymore [1]. In the past the WMF would do 
something with community input. I guess the new strategy is to just give 
the appearance of community input to legitimize a decision. Nice way to 
alienate the volunteers.


Maarten

[1] 
https://lists.wikimedia.org/pipermail/cloud/2020-November/thread.html#1309


On 12-03-2021 19:16, Joaquin Oltra Hernandez wrote:

TLDR:
- Instead of `*.db.svc.eqiad.wmflabs` use `*.db.svc.wikimedia.cloud` 
to use the new replicas

- Quarry will migrate March 23 to use the new cluster
- In a ~month (April 15) the old cluster will start retiring. See 
https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeline 
for more details


Hi everyone,

I'm happy to announce that the new replicas cluster is available for 
use after a few weeks open for testing.


To use the new cluster, you will only need to change the hostname when 
connecting to the databases. Instead of `*.db.svc.eqiad.wmflabs` you 
can use `*.db.svc.wikimedia.cloud`. See:


- 
https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_host_names
- 
https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_can_I_test_the_new_replicas_before_the_switchover?

- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database


This brings us to the timeline updates:
https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeline

- March 2021
  - Announce new cluster domains for all users - March 12
  - Migrate Quarry to use the new cluster - March 23
- April 2021
  - PAWS migration - Estimated first week of April
  - Migrate the old cluster to utilize new replication hosts. 
Replication may stop. - April 15

  - Redirect old hostnames to the new cluster - April 28

Quarry is ready so it will transition first to use the new replicas. 
This will happen on March 23, at which point we will deploy the 
changes and update the docs. We are publishing a note on Tech news but 
if there are other venues where this information would be useful 
please help us spread the word.


PAWS is in the process of being migrated. We estimate it could be 
ready to use the new cluster at the beginning of April, and will 
publish more details as soon as we know.


On April 15 the old cluster will be moved under the new replication 
hosts, at which point there may be side effects and replication may stop.
We recommend you update your code earlier to use the new hostnames to 
verify things are working normally.


Not long after, the old hostnames (`*.db.svc.eqiad.wmflabs`) will be 
redirected to the new ones (`*.db.svc.wikimedia.cloud`), at which 
point the old cluster will effectively be inaccessible.



If you have questions or need help please reply to 
cl...@lists.wikimedia.org , join the 
#wikimedia-cloud IRC channel, or open a ticket in Phabricator with 
the Data-Services tag.


Thanks,

___
Wikimedia Cloud Services announce mailing list
cloud-annou...@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce

___
Wikimedia Cloud Services mailing list
cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,