This WMCS outage is affecting Beta Cluster (aka: deployment-pre VPS project) and our CI build/test servers (aka: the integration project).
Apologies for any downtime and/or slow CI responses while this is being sorted. Greg ----- Forwarded message from Andrew Bogott <abog...@wikimedia.org> ----- > Date: Wed, 13 Feb 2019 07:25:39 -0600 > From: Andrew Bogott <abog...@wikimedia.org> > To: cloud-annou...@lists.wikimedia.org > Subject: [Cloud] [Cloud-announce] VPS hardware failure -- downtime ongoing > Reply-To: cl...@lists.wikimedia.org > > We're currently experiencing a mysterious hareware failure in our datacenter > -- three different SSDs failed overnight, two of them in cloudvirt1018 and > one of them in cloudvirt1024. The VMs on 1018 are down entirely. We may > move those on 1024 to another host shortly in order to guard against > additional drive failure. > > There's some possibility that we will experience permanent data loss on > cloudvirt1018, but everyone is working hard to avoid this. > > The following VMs are on cloudvirt1018: > > > a11y | reading-web-staging > abogott-scapserver | testlabs > af-puppetdb01 | automation-framework > api | openocr > asdf | quotatest > bastion-eqiad1-02 | bastion > clm-test-01 | community-labs-monitoring > compiler1002 | puppet-diffs > cyberbot-exec-iabot-01 | cyberbot > deployment-db03 | deployment-prep > deployment-db04 | deployment-prep > deployment-memc05 | deployment-prep > deployment-pdfrender02 | deployment-prep > deployment-sca01 | deployment-prep > design-lsg3 | design > eventmetrics-dev01 | eventmetrics > fridolin | catgraph > gtirloni-puppetmaster-01 | testlabs > hadoop-master-3 | analytics > ign | ign2commons > integration-castor03 | integration > integration-slave-docker-1017 | integration > integration-slave-docker-1033 | integration > integration-slave-docker-1038 | integration > integration-slave-jessie-1003 | integration > integration-slave-jessie-android | integration > k8s-master-01 | general-k8s > k8s-node-03 | general-k8s > k8s-node-05 | general-k8s > k8s-node-06 | general-k8s > kdc | analytics > labstash-jessie1 | logging > language-mleb-legacy | language > login-test | catgraph > lsg-01 | design > mathosphere | math > mc-clusterA-1 | test-twemproxy > mwoffliner5 | mwoffliner > novaadminmadethis-4 | quotatest > ntp-01 | cloudinfra > ntp-02 | cloudinfra > ogvjs-testing | ogvjs-integration > phragile-pro | phragile > planet-hotdog | planet > pub2 | wikiapiary > puppenmeister | planet > puppet-compiler-v4-other | testlabs > puppet-compiler-v4-tools | testlabs > quarry-beta-01 | quarry > signwriting-swis | signwriting > signwriting-swserver | signwriting > social-tools3 | social-tools > striker-deploy04 | striker > striker-puppet01 | striker > t166878 | otrs > togetherjs | visualeditor > tools-sgebastion-06 | tools > tools-sgeexec-0902 | tools > tools-sgeexec-0903 | tools > tools-sgewebgrid-generic-0901 | tools > tools-sgewebgrid-lighttpd-0901 | tools > ve-font | design > wikibase1 | sciencesource > wikicitevis-prod | wikicitevis > wikifarm | pluggableauth > women-in-red | globaleducation > > > > _______________________________________________ > Wikimedia Cloud Services announce mailing list > cloud-annou...@lists.wikimedia.org (formerly > labs-annou...@lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/cloud-announce > _______________________________________________ > Wikimedia Cloud Services mailing list > cl...@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/cloud ----- End forwarded message ----- -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | Release Team Manager A18D 1138 8E47 FAC8 1C7D | _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l