[Cloud] Re: Tunnel to Trove instance?

2024-06-24 Thread David Caro
-- cloud@lists.wikimedia.org > >>> List information: > >>> https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ > >>> > >> > >> > >> -- > >> Daniel Zahn > >> Site Reliability Engineer > >> _

[Cloud] Re: zsh is slow on login.toolforge.net

2024-04-25 Thread David Caro
f you are using. > > -- > Taavi Väänänen (he/him) > Site Reliability Engineer, Cloud Services > Wikimedia Foundation > ___ > Cloud mailing list -- cloud@lists.wikimedia.org > List information: > https://lists.wikimed

[Cloud] Re: [Cloud-announce] New dev.toolforge.org bastion

2024-04-17 Thread David Caro
; Site Reliability Engineer, Cloud Services > Wikimedia Foundation > ___ > Cloud-announce mailing list -- cloud-annou...@lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/ -- David Caro SRE - Cloud Services Wikimedia Foun

[Cloud-announce] Re: New dev.toolforge.org bastion

2024-04-17 Thread David Caro
; Site Reliability Engineer, Cloud Services > Wikimedia Foundation > ___ > Cloud-announce mailing list -- cloud-announce@lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/ -- David Caro SRE - Cloud Services Wikimedia Foun

[Cloud] [Cloud-announce] Re: [toolforge] Health checks available for webservices

2024-02-29 Thread David Caro
I'm the worst xd The parameter is actually `--health-check-path `. Sorry for the confusion. On 02/29 13:57, David Caro wrote: > > Hi all! > > Good news, we have enabled health checks for all the webservices running on > toolforge. > > There's no action required on you

[Cloud-announce] Re: [toolforge] Health checks available for webservices

2024-02-29 Thread David Caro
I'm the worst xd The parameter is actually `--health-check-path `. Sorry for the confusion. On 02/29 13:57, David Caro wrote: > > Hi all! > > Good news, we have enabled health checks for all the webservices running on > toolforge. > > There's no action required on you

[Cloud] Re: [Cloud-announce] [toolforge] Health checks available for webservices

2024-02-29 Thread David Caro
On 02/29 14:30, Maciej Jaros wrote: > David Caro (2024-02-29 13:57): > > [...] > > > > The most interesting feature though is being able to pass a url to use as > > HTTP > > health check. > > > > To do so you can pass `--health-check-url /path/to/h

[Cloud] [Cloud-announce] [toolforge] Health checks available for webservices

2024-02-29 Thread David Caro
/Portal:Toolforge/Changelog Original task: https://phabricator.wikimedia.org/T341919 Cheers! -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single huma

[Cloud-announce] [toolforge] Health checks available for webservices

2024-02-29 Thread David Caro
/Portal:Toolforge/Changelog Original task: https://phabricator.wikimedia.org/T341919 Cheers! -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single huma

[Cloud] Re: Horizon: pause vs suspend vs shelve?

2024-01-31 Thread David Caro
Hi Roy, I think that what you want is 'shut off', that will turn it off as if the power button was pressed, the OS will halt the machine, then you will turn it on as if you pressed the button and the OS will do the boot as usual. Shelving will save the instance state in disk, so when you

[Cloud] Re: How to run Docker images on Toolforge

2024-01-26 Thread David Caro
Hi! We don't support bringing your own docker image on toolforge, you'll have to use the build service as you mention ( https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service) or one of the existing images (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web ). If you find issues or

[Cloud] Re: [Cloud-announce] Announcing Toolforge Build Service Open Beta

2023-10-24 Thread David Caro
Sorry that was not complete xd, still getting used to the gmail UI On Tue, Oct 24, 2023 at 2:13 PM David Caro wrote: > Hi Maciej! > > Replying inline > > On Mon, Oct 23, 2023 at 9:03 PM Maciej Jaros wrote: > >> Hi >> >> Build tool sounds like an interesting

[Cloud] Re: [Cloud-announce] Announcing Toolforge Build Service Open Beta

2023-10-24 Thread David Caro
Hi Maciej! Replying inline On Mon, Oct 23, 2023 at 9:03 PM Maciej Jaros wrote: > Hi > > Build tool sounds like an interesting idea. What is the intended use case > here? > >1. Using for complex tools that might require complex or too specific >tools on the server side? >2. Using

[Cloud-announce] Re: 2023-10-23 13:00 UTC: Small outage planned for Openstack network cleanup

2023-10-23 Thread David Caro
Done! Everything went as expected :), we had a small network downtime of ~45s, and no side-effects have been detected. If you see anything suspicious please open a task and mention this one T348140 <https://phabricator.wikimedia.org/T348140>. Cheers! On Mon, Oct 23, 2023 at 3:00 PM Davi

[Cloud] [Cloud-announce] Re: 2023-10-23 13:00 UTC: Small outage planned for Openstack network cleanup

2023-10-23 Thread David Caro
Done! Everything went as expected :), we had a small network downtime of ~45s, and no side-effects have been detected. If you see anything suspicious please open a task and mention this one T348140 <https://phabricator.wikimedia.org/T348140>. Cheers! On Mon, Oct 23, 2023 at 3:00 PM Davi

[Cloud] [Cloud-announce] Re: 2023-10-23 13:00 UTC: Small outage planned for Openstack network cleanup

2023-10-23 Thread David Caro
Hi all! As scheduled, this maintenance will be starting, I'll update once we have finished. Cheers! On Tue, Oct 17, 2023 at 4:53 PM David Caro wrote: > Hi! > > There will be a small network interruption next Monday at around 13:00 UTC > as we will be doing some cleanup on the Ope

[Cloud-announce] Re: 2023-10-23 13:00 UTC: Small outage planned for Openstack network cleanup

2023-10-23 Thread David Caro
Hi all! As scheduled, this maintenance will be starting, I'll update once we have finished. Cheers! On Tue, Oct 17, 2023 at 4:53 PM David Caro wrote: > Hi! > > There will be a small network interruption next Monday at around 13:00 UTC > as we will be doing some cleanup on the Ope

[Cloud] [Cloud-announce] 2023-10-23 13:00 UTC: Small outage planned for Openstack network cleanup

2023-10-17 Thread David Caro
will not work during the downtime. We will update on this email thread when we start and when we have finished. This will help stabilize the network and avoid bigger outages in the future, so thanks for your patience! -- David Caro SRE - Cloud Services Wikimedia Foundation <ht

[Cloud-announce] 2023-10-23 13:00 UTC: Small outage planned for Openstack network cleanup

2023-10-17 Thread David Caro
will not work during the downtime. We will update on this email thread when we start and when we have finished. This will help stabilize the network and avoid bigger outages in the future, so thanks for your patience! -- David Caro SRE - Cloud Services Wikimedia Foundation <ht

[Cloud] [Cloud-announce] Re: Current outage on CloudVPS

2023-09-29 Thread David Caro
Good news! All the services should be back up and running! If you still have issues, please ping us on IRC or open a phabricator task. Thanks again for your patience! On Fri, Sep 29, 2023 at 1:11 PM David Caro wrote: > Most of the outage is resolved already, we are finishing up recover

[Cloud-announce] Re: Current outage on CloudVPS

2023-09-29 Thread David Caro
Good news! All the services should be back up and running! If you still have issues, please ping us on IRC or open a phabricator task. Thanks again for your patience! On Fri, Sep 29, 2023 at 1:11 PM David Caro wrote: > Most of the outage is resolved already, we are finishing up recover

[Cloud] [Cloud-announce] Re: Current outage on CloudVPS

2023-09-29 Thread David Caro
, 2023 at 9:31 AM David Caro wrote: > There is an ongoing outage affecting all cloud vps projects (this includes > toolforge and paws) that prevents the machines from getting ip refreshes > (dchp client got uninstalled). > > > We are working on it and the service shoul

[Cloud-announce] Re: Current outage on CloudVPS

2023-09-29 Thread David Caro
, 2023 at 9:31 AM David Caro wrote: > There is an ongoing outage affecting all cloud vps projects (this includes > toolforge and paws) that prevents the machines from getting ip refreshes > (dchp client got uninstalled). > > > We are working on it and the service shoul

[Cloud] [Cloud-announce] Current outage on CloudVPS

2023-09-29 Thread David Caro
There is an ongoing outage affecting all cloud vps projects (this includes toolforge and paws) that prevents the machines from getting ip refreshes (dchp client got uninstalled). We are working on it and the service should be restored soon, will update once everything is up and running. Working

[Cloud-announce] Current outage on CloudVPS

2023-09-29 Thread David Caro
There is an ongoing outage affecting all cloud vps projects (this includes toolforge and paws) that prevents the machines from getting ip refreshes (dchp client got uninstalled). We are working on it and the service should be restored soon, will update once everything is up and running. Working

[Cloud] [Cloud-announce] Re: Switch restart today at 13:00 UTC, no downtime expected

2023-05-15 Thread David Caro
Unfortunately, the tools NFS server was affected, and some processes got hung (see https://phabricator.wikimedia.org/T257945). We are restarting some of the toolforge nodes to get them unstuck, we'll get things back in a fully working state soon. On Mon, 2023-05-15 at 17:34 +0200, David Caro

[Cloud-announce] Re: Switch restart today at 13:00 UTC, no downtime expected

2023-05-15 Thread David Caro
Unfortunately, the tools NFS server was affected, and some processes got hung (see https://phabricator.wikimedia.org/T257945). We are restarting some of the toolforge nodes to get them unstuck, we'll get things back in a fully working state soon. On Mon, 2023-05-15 at 17:34 +0200, David Caro

[Cloud] [Cloud-announce] Re: Switch restart today at 13:00 UTC, no downtime expected

2023-05-15 Thread David Caro
! On Mon, 2023-05-15 at 13:55 +0200, David Caro wrote: > Hi! > > We are restarting a switch[1] today at 13:00 UTC. > > We are moving all the affected VMs to different hypervisors, and we > expect no downtime, though you might experience the servers being a > bit > unrespo

[Cloud-announce] Re: Switch restart today at 13:00 UTC, no downtime expected

2023-05-15 Thread David Caro
! On Mon, 2023-05-15 at 13:55 +0200, David Caro wrote: > Hi! > > We are restarting a switch[1] today at 13:00 UTC. > > We are moving all the affected VMs to different hypervisors, and we > expect no downtime, though you might experience the servers being a > bit > unrespo

[Cloud] [Cloud-announce] Switch restart today at 13:00 UTC, no downtime expected

2023-05-15 Thread David Caro
once it's done. Thanks! [1]https://phabricator.wikimedia.org/T316544 --- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can f

[Cloud-announce] Switch restart today at 13:00 UTC, no downtime expected

2023-05-15 Thread David Caro
once it's done. Thanks! [1]https://phabricator.wikimedia.org/T316544 --- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can f

[Cloud] Re: [Cloud-announce] Re: PAWS nfs cutover 2023-03-14 12:00 UTC

2023-03-14 Thread David Caro
diafoundation.org/> > > > > > -- > > *Vivian Rook (They/Them)* > Site Reliability Engineer > Wikimedia Foundation <https://wikimediafoundation.org/> > ___ > Cloud-announce mailing list -- cloud-annou

[Cloud-announce] Re: PAWS nfs cutover 2023-03-14 12:00 UTC

2023-03-14 Thread David Caro
diafoundation.org/> > > > > > -- > > *Vivian Rook (They/Them)* > Site Reliability Engineer > Wikimedia Foundation <https://wikimediafoundation.org/> > ___ > Cloud-announce mailing list -- cloud-annou

[Cloud] Re: Migrating to a newer VPS image?

2023-01-03 Thread David Caro
ists.wikimedia.org > > List information: > > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ > > > ___ > Cloud mailing list -- cloud@lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ --

[Cloud] Re: Key errors on toolsdb database

2022-12-08 Thread David Caro
Russell Blau wrote: >As it says in the subject line, it is toolsdb. > __ > >From: David Caro >Sent: Thursday, December 8, 2022 11:07 AM >To: Wikimedia Cloud Services general discussion and s

[Cloud] Re: multiple proxies to same cloud vps instance

2022-11-30 Thread David Caro
g > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every

[Cloud] Re: WMCS team work transparency (was: New servers don't show up on grafana-labs.wikimedia.org)

2022-10-06 Thread David Caro
for that, I'll bring the subject to the team for discussion. Otherwise, we have been quite stall with the project. Thanks again for raising this, it helps to get nudged in the right direction  On Thu, 6 Oct 2022, 13:39 Taavi Väänänen, wrote: > On 10/5/22 21:01, David Caro wrote: > > As B

[Cloud] Re: New servers don't show up on grafana-labs.wikimedia.org

2022-10-05 Thread David Caro
> Cloud mailing list -- cloud@lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 8

[Cloud] [Cloud-announce] Re: Current disruptions for VMs/toolforge, disk IO

2022-08-11 Thread David Caro
for your patience and understanding! On 08/11 13:32, David Caro wrote: > Hi everyone! > > TL;DR; > Currently there's a degradation on the service for VMs and anything running > on them (ex. toolforge, quarry, paws, > ...), you might be able to use the services or they might b

[Cloud-announce] Re: Current disruptions for VMs/toolforge, disk IO

2022-08-11 Thread David Caro
for your patience and understanding! On 08/11 13:32, David Caro wrote: > Hi everyone! > > TL;DR; > Currently there's a degradation on the service for VMs and anything running > on them (ex. toolforge, quarry, paws, > ...), you might be able to use the services or they might b

[Cloud] [Cloud-announce] Current disruptions for VMs/toolforge, disk IO

2022-08-11 Thread David Caro
are actively working on this, and will update with any changes. Cheers! -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can f

[Cloud-announce] Current disruptions for VMs/toolforge, disk IO

2022-08-11 Thread David Caro
are actively working on this, and will update with any changes. Cheers! -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can f

[Cloud] Re: Kubernetes-based jobs engine -- how to use Python virtual environments?

2022-04-05 Thread David Caro
Reliability Engineer > > Wikimedia Cloud Services > > Wikimedia Foundation > > > ___ > Cloud mailing list -- cloud@lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ -

[Cloud] Re: Redis eviction stats?

2022-03-21 Thread David Caro
> Cloud mailing list -- cloud@lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1

[Cloud] Re: django with oauth?

2022-02-16 Thread David Caro
I was looking at this a couple months back, and ended up writing the oauth dance myself: https://github.com/david-caro/wm-what/blob/main/wm_what/app.py#L162 Seemed simpler given the lack of maintenance, and given that I just needed a very simple integration (using flask though). On 02/11 22

[Cloud] Re: Storage options: NFS, Cinder, Ceph, Swift?

2021-12-23 Thread David Caro
Hi Sasha, As of right now it's not possible to access the ceph cluster from the Cloud VPS instances directly, that includes Rados Gateways (rgw). That is the reason why we are trying to setup swift on top of it instead, but that's still on the works, and got de-prioitized due to changes in the

[Cloud] Re: Eliminating Debian Stretch in Cloud VPS

2021-12-16 Thread David Caro
ists/cloud.lists.wikimedia.org/ -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge

[Cloud] Re: toolforge down?

2021-11-02 Thread David Caro
The incident has been resolved, all the alerts will start clearing up shortly. If you still see some issues, please ping us here or on IRC (#wikimedia-cloud on libera.chat). Thanks for you patience! On 11/02 13:43, David Caro wrote: > Hi Roy, > > We are currently having a netwo

[Cloud] Re: toolforge down?

2021-11-02 Thread David Caro
information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human

[Cloud] Re: Hibernating a VM?

2021-06-07 Thread David Caro
ast one because in my exploring, I accidentally clicked > it, but other than seeing a "shelving" progress bar appear, it doesn't seem > to have actually done anything. My instance is still running normally. > Fingers crossed :-) > > > On Jun 7, 2021, at 9:20 AM, David Ca

[Cloud] Re: Hibernating a VM?

2021-06-07 Thread David Caro
> Cloud mailing list -- cloud@lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2

Re: [Cloud-announce] [Cloud] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread David Caro
__ > > Wikimedia Cloud Services mailing list > > cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) > > https://lists.wikimedia.org/mailman/listinfo/cloud > > ___ > Wikimedia Cloud Services mailing list >

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread David Caro
__ > > Wikimedia Cloud Services mailing list > > Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) > > https://lists.wikimedia.org/mailman/listinfo/cloud > > ___ > Wikimedia Cloud Services mailing list >

Re: [Cloud-announce] Toolforge kubernetes maintenance today 2020-12-10 @ 15:30 UTC

2020-12-10 Thread David Caro
vices > Wikimedia Foundation > > ___ > Wikimedia Cloud Services announce mailing list > Cloud-announce@lists.wikimedia.org (formerly > labs-annou...@lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Re: [Cloud] [Cloud-announce] Toolforge kubernetes maintenance today 2020-12-10 @ 15:30 UTC

2020-12-10 Thread David Caro
Services > Wikimedia Foundation > > ___ > Wikimedia Cloud Services announce mailing list > cloud-annou...@lists.wikimedia.org (formerly > labs-annou...@lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

2020-11-19 Thread David Caro
gt; https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication > >>> [3]: > >>> https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign > >>> > >>> -- > >>> Joaquin Oltra Hernandez > >&