Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

[AntiCompositeNumber]

Most cross-db JOINs can be recreated using two queries and an external
tool to filter the results. However, there are some queries that would
be simply impractical due to the large amount of data involved, and
the query for overlapping local and Commons images is one of them.
There are basically two ways to recreate the query: re-implement the
inner join or re-implement a semi-join subquery.

Recreating a JOIN is conceptually very simple: get two lists and
compare them. However, there are 67,034 files on fawiki, 891,286 files
on enwiki, and 65,559,375 files on Commons. Simply joining by name
would be impossible -- MariaDB would time out a few hundred times
before returning all that data, and even if it did, storing those
lists even as efficiently as possible would be quite the memory hog.
So the query would have to be paginated. The only common identifier we
have is the file name, and since the letters in the names aren't
evenly distributed, paginating wouldn't exactly be fun.
The other option is implementing the Commons lookup like a semi-join
subquery. Iterate over the local data, paginating any way you want.
Then, for every item, query the Commons database for that title. Of
course, we're now making a million requests to the database, which
isn't going to be very fast simply due to network delays. We could be
a little nicer and group a bunch of titles together in the query,
which will probably get us down from a million queries to fifty
thousand or so. Of course, this all gets more complicated if you want
a query more complex than SELECT enwiki_p.img_title FROM
enwiki_p.image JOIN commonswiki_p.image ON enwiki_p.img_title =
commonswiki_p.img_title;

I understand the system engineering reasons for this change, but I
think it's worth underscoring exactly how disruptive it will be for
the queries that depended on this functionality. I'm certainly no
expert, but I'm willing to help wrap queries in Python until they
start working again.

ACN

On Tue, Nov 10, 2020 at 8:48 PM Huji Lee  wrote:
>
> Cross-wiki JOINS are used by some of the queries we run regularly for fawiki. 
> One of those queries looks for articles that don't have an image in their 
> infobox in fawiki, but do have one on enwiki, so that we can use/import that 
> image. Another one JOINs fawiki data with commons data to look for redundant 
> images. Yet another one, looks for articles that all use an image that 
> doesn't exist (for cleanup purposes) but needs to join with commons db 
> because the referenced file might exist there. Lastly, we have a report that 
> looks for fair use images on fawiki that had the same name as an image on 
> enwiki where the enwiki copy was deleted; this usually indicates in improper 
> application of fair use, and enwiki -- due to its larger community -- finds 
> and deletes these faster than we could on fawiki.
>
> There may be other cases I am unaware of. The point is, losing the cross-wiki 
> JOIN capability can make some of the above tasks really difficult or 
> completely impossible.
>
> On Tue, Nov 10, 2020 at 3:27 PM Joaquin Oltra Hernandez 
>  wrote:
>>
>> TLDR: Wiki Replicas' architecture is being redesigned for stability and 
>> performance. Cross database JOINs will not be available and a host 
>> connection will only allow querying its associated DB. See [1] for more 
>> details.
>>
>> Hi!
>>
>> In the interest of making and keeping Wiki Replicas a stable and performant 
>> service, a new backend architecture is needed. There is some impact in the 
>> features and usage patterns.
>>
>> What should I do? To avoid breaking changes, you can start making the 
>> following changes *now*:
>> - Update existing tools to ensure queries are executed against the proper 
>> database connection
>>   - Eg: If you want to query the `eswiki_p` DB, you must connect to the 
>> `eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to 
>> enwiki or other hosts
>> - Check your existing tools and services queries for cross database JOINs, 
>> rewrite the joins in application code
>>   - Eg: If you are doing a join across databases, for example joining 
>> `enwiki_p` and `eswiki_p`, you will need to query them separately, and 
>> filter the results of the separate queries in the code
>>
>> Timeline:
>> - November - December: Early adopter testing
>> - January 2021: Existing and new systems online, transition period starts
>> - February 2021: Old hardware is decommissioned
>>
>> We need your help
>> - If you would like to beta test the new architecture, please let us know 
>> and we will reach out to you soon
>> - Sharing examples / descriptions of how a tool or service was updated, 
>> writing a common solution or some example code others can utilize and 
>> reference, helping others on IRC and the mailing lists
>>
>> If you have questions or need help adapting your code or queries, please 
>> contact us [2], or write on the talk page [3].
>>
>> We will be sending reminders, and more specific examples 

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

[MusikAnimal]

Got it. The https://noc.wikimedia.org/conf/dblists/ lists are plenty fast
and easy enough to parse. I'll just cache that. It would be neat if we
could rely on the slice specified in meta_p in the future, as in my case we
have to query meta_p.wiki regardless, but not a big deal :)

Thank you! I think I have enough information to move forward.

~ MA

On Tue, Nov 10, 2020 at 5:13 PM Brooke Storm  wrote:

> Yes, you might be able to use meta_p.wiki table. However, when wikis are
> moved between sections, nothing updates the meta_p.wiki table at this time.
> Requests to noc.wikimedia.org are accurate and up to date, as far as I
> know. We only update meta_p when we add the wiki (at least that’s how it is
> now).  Also, the DNS gets synced and updated every time we run the script,
> so it is usually up-to-date.  You could try meta_p.wiki and fall back to
> DNS or noc.wikimedia.org if that fails, perhaps?  Meta_p is expected to
> be on s7 in the new design.
>
> Brooke Storm
> Staff SRE
> Wikimedia Cloud Services
> bst...@wikimedia.org
> IRC: bstorm
>
> On Nov 10, 2020, at 3:01 PM, MusikAnimal  wrote:
>
> Ah yes, 8 tunnels is more than manageable. The `slice` column in the
> meta_p.wiki table is the one we need to connect to for said wiki, right? So
> in theory, I always have SSH tunnels open for every slice, and the first
> thing I do is check meta_p.wiki for the given wiki, then I know which of
> those s1-s8 connections to use? So I really only need 8 connections (even
> in production). Maybe not what you would recommend for every tool, rather
> just the "global" ones facing this specific issue.
>
> > Can't you just tunnel to the login server and connect by hostname from
> there?
>
> Hmm I'm not sure I follow. Right now I SHH to login.toolforge.org, but
> with "-L 4711:enwiki.analytics.db.svc.eqiad.wmflabs:3306" for port
> forwarding from my local mysql to the remote. It sounds like instead I need
> to tunnel to s1-s8, and use the correct one based on the desired database.
>
> ~ MA
>
> On Tue, Nov 10, 2020 at 4:32 PM Brooke Storm  wrote:
>
>> Hi MA,
>> You could still accomplish the local environment you are describing by
>> using 8 ssh tunnels. All the database name DNS aliases go reference the
>> section names, eventually (s1, s2, s3, s4 in the form of
>> s1.analytics.db.svc.eqiad.wmflabs, etc.). An app could be written to
>> connect to the correct section instead of the database if you are doing
>> that kind of thing, but you’ll either need to make requests to
>> https://noc.wikimedia.org/conf/dblists/s.dblist like
>> https://noc.wikimedia.org/conf/dblists/s4.dblist and map things out from
>> there or perhaps check DNS for the database name and look up the “s#”
>> record from there (which is currently possible in Lua, and I can provide an
>> example of how I did it in that language).
>>
>> A mediawiki config checkout would also work besides what can be gleaned
>> from noc.wikimedia.org.
>>
>> We can try to document some examples of how you might do it either way.
>> I’m sure it is non-trivial, but 8 tunnels is more workable than 900, at
>> least.
>>
>> Routing by reading the queries on the fly is quite tricky. The closest
>> I’ve seen ready-made tools come to that is ProxySQL, and that focuses on
>> sharding, which is not exactly the same thing.
>>
>> Brooke Storm
>> Staff SRE
>> Wikimedia Cloud Services
>> bst...@wikimedia.org
>> IRC: bstorm
>>
>> On Nov 10, 2020, at 2:13 PM, MusikAnimal  wrote:
>>
>> Hi! Most tools query just a single db at a time, so I don't think this
>> will be a massive problem. However some such as Global
>> Contribs[0] and GUC[1] can theoretically query all of them from a single
>> request. Creating new connections on-the-fly seems doable in production,
>> the issue is how to work on these tools in a local environment. Currently
>> the recommendation is to use a SSH tunnel to the desired host,[2] such
>> as enwiki.analytics.db.svc.eqiad.wmflabs. Surely we can't do this same port
>> forwarding for 900+ connections.
>>
>> Any ideas? Perhaps there's some way to make a host that automatically
>> forwards to the correct one, solely for developer use? Or will development
>> of such global tools need to happen in the Cloud Services environment?
>>
>> ~ MA
>>
>> [0] https://xtools.wmflabs.org/globalcontribs
>> [1] https://guc.toolforge.org/
>> [2]
>> https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#SSH_tunneling_for_local_testing_which_makes_use_of_Wiki_Replica_databases
>>
>> On Tue, Nov 10, 2020 at 3:26 PM Joaquin Oltra Hernandez <
>> jhernan...@wikimedia.org> wrote:
>>
>>> TLDR: Wiki Replicas' architecture is being redesigned for stability and
>>> performance. Cross database JOINs will not be available and a host
>>> connection will only allow querying its associated DB. See [1]
>>> 
>>> for more details.
>>>
>>> Hi!
>>>
>>> In the interest of making and keeping Wiki Replicas a stable and
>>> performant 

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

[Gergo Tisza]

On Tue, Nov 10, 2020 at 1:15 PM MusikAnimal  wrote:

> Hi! Most tools query just a single db at a time, so I don't think this
> will be a massive problem. However some such as Global
> Contribs[0] and GUC[1] can theoretically query all of them from a single
> request. Creating new connections on-the-fly seems doable in production,
> the issue is how to work on these tools in a local environment. Currently
> the recommendation is to use a SSH tunnel to the desired host,[2] such
> as enwiki.analytics.db.svc.eqiad.wmflabs. Surely we can't do this same port
> forwarding for 900+ connections.
>
> Any ideas? Perhaps there's some way to make a host that automatically
> forwards to the correct one, solely for developer use? Or will development
> of such global tools need to happen in the Cloud Services environment?
>

Can't you just tunnel to the login server and connect by hostname from
there?
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

[Brooke Storm]

Hi MA,
You could still accomplish the local environment you are describing by using 8 
ssh tunnels. All the database name DNS aliases go reference the section names, 
eventually (s1, s2, s3, s4 in the form of s1.analytics.db.svc.eqiad.wmflabs, 
etc.). An app could be written to connect to the correct section instead of the 
database if you are doing that kind of thing, but you’ll either need to make 
requests to https://noc.wikimedia.org/conf/dblists/s.dblist 
 like 
https://noc.wikimedia.org/conf/dblists/s4.dblist 
 and map things out from 
there or perhaps check DNS for the database name and look up the “s#” record 
from there (which is currently possible in Lua, and I can provide an example of 
how I did it in that language).

A mediawiki config checkout would also work besides what can be gleaned from 
noc.wikimedia.org .

We can try to document some examples of how you might do it either way. I’m 
sure it is non-trivial, but 8 tunnels is more workable than 900, at least.

Routing by reading the queries on the fly is quite tricky. The closest I’ve 
seen ready-made tools come to that is ProxySQL, and that focuses on sharding, 
which is not exactly the same thing.

Brooke Storm
Staff SRE
Wikimedia Cloud Services
bst...@wikimedia.org 
IRC: bstorm

> On Nov 10, 2020, at 2:13 PM, MusikAnimal  wrote:
> 
> Hi! Most tools query just a single db at a time, so I don't think this will 
> be a massive problem. However some such as Global Contribs[0] and GUC[1] can 
> theoretically query all of them from a single request. Creating new 
> connections on-the-fly seems doable in production, the issue is how to work 
> on these tools in a local environment. Currently the recommendation is to use 
> a SSH tunnel to the desired host,[2] such as 
> enwiki.analytics.db.svc.eqiad.wmflabs. Surely we can't do this same port 
> forwarding for 900+ connections.
> 
> Any ideas? Perhaps there's some way to make a host that automatically 
> forwards to the correct one, solely for developer use? Or will development of 
> such global tools need to happen in the Cloud Services environment?
> 
> ~ MA
> 
> [0] https://xtools.wmflabs.org/globalcontribs 
> 
> [1] https://guc.toolforge.org/ 
> [2] 
> https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#SSH_tunneling_for_local_testing_which_makes_use_of_Wiki_Replica_databases
>  
> 
> On Tue, Nov 10, 2020 at 3:26 PM Joaquin Oltra Hernandez 
> mailto:jhernan...@wikimedia.org>> wrote:
> TLDR: Wiki Replicas' architecture is being redesigned for stability and 
> performance. Cross database JOINs will not be available and a host connection 
> will only allow querying its associated DB. See [1] 
>  for 
> more details.
> 
> Hi!
> 
> In the interest of making and keeping Wiki Replicas a stable and performant 
> service, a new backend architecture is needed. There is some impact in the 
> features and usage patterns.
> 
> What should I do? To avoid breaking changes, you can start making the 
> following changes *now*:
> - Update existing tools to ensure queries are executed against the proper 
> database connection
>   - Eg: If you want to query the `eswiki_p` DB, you must connect to the 
> `eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to 
> enwiki or other hosts
> - Check your existing tools and services queries for cross database JOINs, 
> rewrite the joins in application code
>   - Eg: If you are doing a join across databases, for example joining 
> `enwiki_p` and `eswiki_p`, you will need to query them separately, and filter 
> the results of the separate queries in the code
> 
> Timeline:
> - November - December: Early adopter testing
> - January 2021: Existing and new systems online, transition period starts
> - February 2021: Old hardware is decommissioned
> 
> We need your help
> - If you would like to beta test the new architecture, please let us know and 
> we will reach out to you soon
> - Sharing examples / descriptions of how a tool or service was updated, 
> writing a common solution or some example code others can utilize and 
> reference, helping others on IRC and the mailing lists
> 
> If you have questions or need help adapting your code or queries, please 
> contact us [2] 
> , or 
> write on the talk page [3] 
> .
> 
> We will be sending reminders, and more specific examples of the changes via 
> email and on the wiki page. For more information see [1] 
> 

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

[MusikAnimal]

Hi! Most tools query just a single db at a time, so I don't think this will
be a massive problem. However some such as Global
Contribs[0] and GUC[1] can theoretically query all of them from a single
request. Creating new connections on-the-fly seems doable in production,
the issue is how to work on these tools in a local environment. Currently
the recommendation is to use a SSH tunnel to the desired host,[2] such
as enwiki.analytics.db.svc.eqiad.wmflabs. Surely we can't do this same port
forwarding for 900+ connections.

Any ideas? Perhaps there's some way to make a host that automatically
forwards to the correct one, solely for developer use? Or will development
of such global tools need to happen in the Cloud Services environment?

~ MA

[0] https://xtools.wmflabs.org/globalcontribs
[1] https://guc.toolforge.org/
[2]
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#SSH_tunneling_for_local_testing_which_makes_use_of_Wiki_Replica_databases

On Tue, Nov 10, 2020 at 3:26 PM Joaquin Oltra Hernandez <
jhernan...@wikimedia.org> wrote:

> TLDR: Wiki Replicas' architecture is being redesigned for stability and
> performance. Cross database JOINs will not be available and a host
> connection will only allow querying its associated DB. See [1]
> 
> for more details.
>
> Hi!
>
> In the interest of making and keeping Wiki Replicas a stable and
> performant service, a new backend architecture is needed. There is some
> impact in the features and usage patterns.
>
> What should I do? To avoid breaking changes, you can start making the
> following changes *now*:
> - Update existing tools to ensure queries are executed against the proper
> database connection
>   - Eg: If you want to query the `eswiki_p` DB, you must connect to the
> `eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to
> enwiki or other hosts
> - Check your existing tools and services queries for cross database JOINs,
> rewrite the joins in application code
>   - Eg: If you are doing a join across databases, for example joining
> `enwiki_p` and `eswiki_p`, you will need to query them separately, and
> filter the results of the separate queries in the code
>
> Timeline:
> - November - December: Early adopter testing
> - January 2021: Existing and new systems online, transition period starts
> - February 2021: Old hardware is decommissioned
>
> We need your help
> - If you would like to beta test the new architecture, please let us know
> and we will reach out to you soon
> - Sharing examples / descriptions of how a tool or service was updated,
> writing a common solution or some example code others can utilize and
> reference, helping others on IRC and the mailing lists
>
> If you have questions or need help adapting your code or queries, please
> contact us [2]
> ,
> or write on the talk page [3]
> 
> .
>
> We will be sending reminders, and more specific examples of the changes
> via email and on the wiki page. For more information see [1]
> .
>
> [1]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign
> [2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
> [3]:
> https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign
>
> --
> Joaquin Oltra Hernandez
> Developer Advocate - Wikimedia Foundation
> ___
> Wikimedia Cloud Services announce mailing list
> cloud-annou...@lists.wikimedia.org (formerly
> labs-annou...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud-announce
>
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

[Joaquin Oltra Hernandez]

TLDR: Wiki Replicas' architecture is being redesigned for stability and
performance. Cross database JOINs will not be available and a host
connection will only allow querying its associated DB. See [1]
 for
more details.

Hi!

In the interest of making and keeping Wiki Replicas a stable and performant
service, a new backend architecture is needed. There is some impact in the
features and usage patterns.

What should I do? To avoid breaking changes, you can start making the
following changes *now*:
- Update existing tools to ensure queries are executed against the proper
database connection
  - Eg: If you want to query the `eswiki_p` DB, you must connect to the
`eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to
enwiki or other hosts
- Check your existing tools and services queries for cross database JOINs,
rewrite the joins in application code
  - Eg: If you are doing a join across databases, for example joining
`enwiki_p` and `eswiki_p`, you will need to query them separately, and
filter the results of the separate queries in the code

Timeline:
- November - December: Early adopter testing
- January 2021: Existing and new systems online, transition period starts
- February 2021: Old hardware is decommissioned

We need your help
- If you would like to beta test the new architecture, please let us know
and we will reach out to you soon
- Sharing examples / descriptions of how a tool or service was updated,
writing a common solution or some example code others can utilize and
reference, helping others on IRC and the mailing lists

If you have questions or need help adapting your code or queries, please
contact us [2]
, or
write on the talk page [3]
.

We will be sending reminders, and more specific examples of the changes via
email and on the wiki page. For more information see [1]
.

[1]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign
[2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
[3]:
https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign

-- 
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation
___
Wikimedia Cloud Services announce mailing list
cloud-annou...@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud-announce] Wiki Replicas 2020 Redesign

[Joaquin Oltra Hernandez]

TLDR: Wiki Replicas' architecture is being redesigned for stability and
performance. Cross database JOINs will not be available and a host
connection will only allow querying its associated DB. See [1]
 for
more details.

Hi!

In the interest of making and keeping Wiki Replicas a stable and performant
service, a new backend architecture is needed. There is some impact in the
features and usage patterns.

What should I do? To avoid breaking changes, you can start making the
following changes *now*:
- Update existing tools to ensure queries are executed against the proper
database connection
  - Eg: If you want to query the `eswiki_p` DB, you must connect to the
`eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to
enwiki or other hosts
- Check your existing tools and services queries for cross database JOINs,
rewrite the joins in application code
  - Eg: If you are doing a join across databases, for example joining
`enwiki_p` and `eswiki_p`, you will need to query them separately, and
filter the results of the separate queries in the code

Timeline:
- November - December: Early adopter testing
- January 2021: Existing and new systems online, transition period starts
- February 2021: Old hardware is decommissioned

We need your help
- If you would like to beta test the new architecture, please let us know
and we will reach out to you soon
- Sharing examples / descriptions of how a tool or service was updated,
writing a common solution or some example code others can utilize and
reference, helping others on IRC and the mailing lists

If you have questions or need help adapting your code or queries, please
contact us [2]
, or
write on the talk page [3]
.

We will be sending reminders, and more specific examples of the changes via
email and on the wiki page. For more information see [1]
.

[1]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign
[2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
[3]:
https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign

-- 
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation
___
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime

[Brooke Storm]

This will be happening in around 10 minutes. ToolsDB will be read-only until we 
can get a consistent dump to rebuild replication.

Brooke Storm
Staff SRE
Wikimedia Cloud Services
bst...@wikimedia.org 
IRC: bstorm

> On Nov 6, 2020, at 12:12 PM, Brooke Storm  wrote:
> 
> The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS 
> has tried to restore replication of data, but that has been unsuccessful so 
> far including doing a dump to rebuild replication without downtime.
> At this point, we have a new server waiting to become the replica, but to 
> start the replication process, we need to set the database to read-only for a 
> full dump. This could easily take more than an hour. During that entire time, 
> the database will be read-only.
> 
> We will begin at 1600 UTC and finish when it is done. The database is quite 
> large, but, with it in read-only mode, I hope the backup will not take 
> terribly long.
> 
> Please see https://phabricator.wikimedia.org/T266587 
>  for additional information.
> 
> Brooke Storm
> Staff SRE
> Wikimedia Cloud Services
> bst...@wikimedia.org 
> IRC: bstorm
> 



signature.asc
Description: Message signed with OpenPGP
___
Wikimedia Cloud Services announce mailing list
cloud-annou...@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime

[Brooke Storm]

This will be happening in around 10 minutes. ToolsDB will be read-only until we 
can get a consistent dump to rebuild replication.

Brooke Storm
Staff SRE
Wikimedia Cloud Services
bst...@wikimedia.org 
IRC: bstorm

> On Nov 6, 2020, at 12:12 PM, Brooke Storm  wrote:
> 
> The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS 
> has tried to restore replication of data, but that has been unsuccessful so 
> far including doing a dump to rebuild replication without downtime.
> At this point, we have a new server waiting to become the replica, but to 
> start the replication process, we need to set the database to read-only for a 
> full dump. This could easily take more than an hour. During that entire time, 
> the database will be read-only.
> 
> We will begin at 1600 UTC and finish when it is done. The database is quite 
> large, but, with it in read-only mode, I hope the backup will not take 
> terribly long.
> 
> Please see https://phabricator.wikimedia.org/T266587 
>  for additional information.
> 
> Brooke Storm
> Staff SRE
> Wikimedia Cloud Services
> bst...@wikimedia.org 
> IRC: bstorm
> 



signature.asc
Description: Message signed with OpenPGP
___
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce