[Wikitech-l] Re: March 2023 Datacenter Switchover

2023-02-21 Thread Clément Goubert
Dear Wikitechians, I would like to remind you that the datacenter switchover will happen on *Wednesday March 1st* starting at *14:00 UTC.* Please refer to the original email for any additional information. As always, you can reach out to me directly or the SRE team in #wikimedia-sre on IRC with a

[Wikitech-l] Re: March 2023 Datacenter Switchover

2023-03-01 Thread Clément Goubert
Dear Wikitechians, Dear colleagues, The switchover process requires a *brief read-only period for all Foundation-hosted wikis*, which started at *14:00 UTC on Wednesday March 1st*, and lasted *119 seconds*. All our public and private wikis continued to be available for reading as usual. Users saw

[Wikitech-l] Re: March 2023 Datacenter Switchover

2023-03-01 Thread Dušan Kreheľ
Clément Goubert and everybody, I analyzed https://stream.wikimedia.org/v2/stream/recentchange and i have the another results. Last change (before migration): 2023-03-01T14:00:30 First change (after migration):2023-03-01T14:02:05 Result: Down time (14:00:31 to 14:02:05) is 94s. I think that a

[Wikitech-l] Re: March 2023 Datacenter Switchover

2023-03-01 Thread Amir Sarabadani
It's a bit complicated. When SRE sets the read-only mark, they start counting from that time and it starts propagating which takes a while to be actually shown to all users but some users might still see the RO error while some actual writes are happening somewhere else because the cache is not inv

[Wikitech-l] Re: March 2023 Datacenter Switchover

2023-03-01 Thread Giuseppe Lavagetto
Specifically: if we measure read only time as "an editor can't start an edit because wikis are read only", then the read-only time is 119s; if we measure it by the last timestamp of an edit being saved, that's 94 seconds. As Amir explained, we leave some room for propagation of the MediaWiki rea

[Wikitech-l] Re: March 2023 Datacenter Switchover

2023-03-01 Thread Igal Khitron
Hello. Is the new files bug happening because of the database switch? And if it is, do you fix it? The story: If you reupload file, local or on commons, the new version does not create thumb, and the old one is shown in articles. The only way to see a new version is opening the file in media: names

[Wikitech-l] Re: March 2023 Datacenter Switchover

2023-03-01 Thread Brian Wolff
You should probably just file a bug. Its certainly plausible it had something to do with data center switch, but it could just as equally be unrelated. It requires someone to investigate what part of the system failed (job queue? Varnish? Swift?) which would then lead to a root cause. Its pretty i

[Wikitech-l] Re: March 2023 Datacenter Switchover

2023-03-01 Thread Igal Khitron
I see. Very well, I'll open a task. Thank you. Igal בתאריך יום ד׳, 1 במרץ 2023, 22:07, מאת Brian Wolff ‏: > You should probably just file a bug. > > Its certainly plausible it had something to do with data center switch, > but it could just as equally be unrelated. It requires someone to > invest

[Wikitech-l] Re: March 2023 Datacenter Switchover

2023-04-20 Thread Szymon Grabarczuk
Hello everyone, Just as Clément wrote in the first message, on *Wednesday April 26*, there will be a switchover from our secondary data center in Texas back to the primary data center in Virginia. That day, starting at *14:00 UTC*, there will be a brief (just a few minutes) read-only period for al