Re: [Cloud-announce] [Cloud] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread Andrew Bogott 
There's still a bit of upgrading happening but the recent network 
interruption should now be resolved. We're about to put Horizon back 
online so things should be back to normal shortly.  As always, some 
tools may have failed to recover from the network interruption so you're 
encouraged to check on the state of things.


Sorry for the interruption!

-Andrew



On 4/27/21 10:37 AM, David Caro wrote:

Yep, it's related to the upgrade, working on it, thanks for your patience!

Will update shortly.

On 04/27 11:34, Roy Smith wrote:

I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ 
 and https://spi-tools-dev.toolforge.org/ 
.  Also:

ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable

Could the VPS work be causing this?



On Apr 27, 2021, at 10:03 AM, Andrew Bogott  wrote:

Reminder: this is starting shortly.

On 4/27/21 12:48 AM, Andrew Bogott wrote:

We will be upgrading the Cloud-VPS OpenStack install later today beginning at 
14:00 UTC (7:00 AM Pacific time).

The total upgrade should take 60-90 minutes.  During the upgrade period Horizon 
will be disabled.  There may also be brief network interruptions as we restart 
router services.

-Andrew + the WMCS Team




___
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
___
Wikimedia Cloud Services mailing list
cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

___
Wikimedia Cloud Services mailing list
cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud



___
Wikimedia Cloud Services mailing list
cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud



___
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Re: [Cloud-announce] [Cloud] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread David Caro 

Yep, it's related to the upgrade, working on it, thanks for your patience!

Will update shortly.

On 04/27 11:34, Roy Smith wrote:
> I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ 
>  and https://spi-tools-dev.toolforge.org/ 
> .  Also:
> 
> ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable
> 
> Could the VPS work be causing this?
> 
> 
> > On Apr 27, 2021, at 10:03 AM, Andrew Bogott  wrote:
> > 
> > Reminder: this is starting shortly.
> > 
> > On 4/27/21 12:48 AM, Andrew Bogott wrote:
> >> We will be upgrading the Cloud-VPS OpenStack install later today beginning 
> >> at 14:00 UTC (7:00 AM Pacific time).
> >> 
> >> The total upgrade should take 60-90 minutes.  During the upgrade period 
> >> Horizon will be disabled.  There may also be brief network interruptions 
> >> as we restart router services.
> >> 
> >> -Andrew + the WMCS Team
> >> 
> >> 
> > 
> > 
> > ___
> > Wikimedia Cloud Services announce mailing list
> > Cloud-announce@lists.wikimedia.org (formerly 
> > labs-annou...@lists.wikimedia.org)
> > https://lists.wikimedia.org/mailman/listinfo/cloud-announce
> > ___
> > Wikimedia Cloud Services mailing list
> > cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> > https://lists.wikimedia.org/mailman/listinfo/cloud
> 

> ___
> Wikimedia Cloud Services mailing list
> cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud


-- 
David Caro
SRE - Cloud Services
Wikimedia Foundation 
PGP Signature: 7180 83A2 AC8B 314F B4CE  1171 4071 C7E1 D262 69C3

"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."


signature.asc
Description: PGP signature
___
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Re: [Cloud-announce] [Cloud] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread Roy Smith 
I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ 
 and https://spi-tools-dev.toolforge.org/ 
.  Also:

ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable

Could the VPS work be causing this?


> On Apr 27, 2021, at 10:03 AM, Andrew Bogott  wrote:
> 
> Reminder: this is starting shortly.
> 
> On 4/27/21 12:48 AM, Andrew Bogott wrote:
>> We will be upgrading the Cloud-VPS OpenStack install later today beginning 
>> at 14:00 UTC (7:00 AM Pacific time).
>> 
>> The total upgrade should take 60-90 minutes.  During the upgrade period 
>> Horizon will be disabled.  There may also be brief network interruptions as 
>> we restart router services.
>> 
>> -Andrew + the WMCS Team
>> 
>> 
> 
> 
> ___
> Wikimedia Cloud Services announce mailing list
> Cloud-announce@lists.wikimedia.org (formerly 
> labs-annou...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud-announce
> ___
> Wikimedia Cloud Services mailing list
> cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud

___
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Re: [Cloud-announce] [Cloud] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread Brooke Storm 
Toolforge NFS services are not accessible to the VMs. Something went wrong 
during the upgrade and the team is working to restore services as quickly as 
possible.

I have shut off the bastions to protect their local disks.

Brooke Storm
Staff SRE
Wikimedia Cloud Services
bst...@wikimedia.org



> On Apr 27, 2021, at 8:34 AM, Roy Smith  wrote:
> 
> I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ 
>  and https://spi-tools-dev.toolforge.org/ 
> .  Also:
> 
> ssh: connect to host dev.tools.wmflabs.org  
> port 22: Network is unreachable
> 
> Could the VPS work be causing this?
> 
> 
>> On Apr 27, 2021, at 10:03 AM, Andrew Bogott > > wrote:
>> 
>> Reminder: this is starting shortly.
>> 
>> On 4/27/21 12:48 AM, Andrew Bogott wrote:
>>> We will be upgrading the Cloud-VPS OpenStack install later today beginning 
>>> at 14:00 UTC (7:00 AM Pacific time).
>>> 
>>> The total upgrade should take 60-90 minutes.  During the upgrade period 
>>> Horizon will be disabled.  There may also be brief network interruptions as 
>>> we restart router services.
>>> 
>>> -Andrew + the WMCS Team
>>> 
>>> 
>> 
>> 
>> ___
>> Wikimedia Cloud Services announce mailing list
>> Cloud-announce@lists.wikimedia.org 
>>  (formerly 
>> labs-annou...@lists.wikimedia.org )
>> https://lists.wikimedia.org/mailman/listinfo/cloud-announce 
>> 
>> ___
>> Wikimedia Cloud Services mailing list
>> cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
> 
> ___
> Wikimedia Cloud Services mailing list
> cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud

___
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce