Hi all, We've indeed had a total site outage for roughly 30 minutes. We're still collecting all data, but we've tracked down the cause to multiple cascading issues including loss of power to a critical SPOF network switch and HHVM MediaWiki application servers getting blocked due to multiple unoptimal timeout settings. We'll post a full incident report soon, and work to correct the underlying issues as soon as possible.
Our apologies, On Thu, Feb 5, 2015 at 7:03 PM, Guillaume Paumier <gpaum...@wikimedia.org> wrote: > Hi, > > Le jeudi 5 février 2015, 09:58:01 George Herbert a écrit : > > I saw a WMF tweet of a site outage (network?) around 9:30am Pacific > time, by > > the time I could check now things seem ok on en > > Sites are mostly back up but there are still issues with login, so the Ops > team hasn't had time to write a postmortem yet. > > -- > Guillaume Paumier > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > -- Mark Bergsma <m...@wikimedia.org> Lead Operations Architect Director of Technical Operations Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
